September was a busy month. In addition to continued fascination over art generation with DALL-E and friends, and the questions they pose for intellectual property, we see interesting things happening with machine learning for low-powered processors: using attention, mechanisms, along with a new microcontroller that can run for a week on a single AA battery. In other parts of the technical universe, “platform engineering” has been proposed as an alternative to both DevOps and SRE. We’ve seen demonstrations of SQL injection-like attacks against GPT-3; and companies including Starbucks, Chipotle, and Universal Studios are offering NFT-based loyalty programs. (In addition to a Chipotle’s steak grilling demo in the Metaverse.)
Artificial Intelligence
- Facebook/Meta ups the ante on AI-generated images: they have a system that creates short videos from a natural language description. Videos are currently limited to five seconds. It isn’t open to the public.
- Transformers, which have a key to the progress in natural language processing, are now being adapted for work in computer vision, displaying convolutional neural networks.
- A group of researchers are talking about bringing attention mechanisms to resource-constrained TinyML applications. Attention mechanisms are the central innovation that led to language tools like GPT-3. Low power attention could revolutionize embedded AI applications.
- AGENT is a new benchmark for “common sense” in AI. It consists of a series of 3D animations. An AI model has to rate the videos as “surprising” or “expected.” To score highly, the model needs to demonstrate a human-like ability to plan, in addition to understanding concepts like basic physics.
- Whisper is a new speech-to-text AI model from OpenAI. Its accuracy is impressive and, unlike other OpenAI products, it is open source.
- Google’s Sparrow is an experimental AI chatbot that has been trained not to generate “dangerous” replies (ranging from hate speech to financial advice and claims of sentience). It is far from perfect, but it appears to be a significant improvement over current chat technology.
- Have I been trained is a web application that searches for specific images in the LAION-5B data set, which was used to train several image generation models. You can search using images or text. It’s useful for discovering whether your artwork or photos were used in training.
- Art generated by AI tools like Midjourney and Stable Diffusion is starting to appear on stock photography web sites. Getty Images has banned AI-generated content because they are concerned about copyright violations.
- A new model for analyzing chest x-ray images learns from natural language medical reports written when the image was taken, rather than images labeled after the fact. Its accuracy is roughly equivalent to human radiologists.
- Amodal panoptic segmentation is a new vision algorithm that allows systems to identify objects that are partially obscured by objects in front. This could be an important technology for improving autonomous vehicles’ ability to identify pedestrians successfully.
- Huggingface has released a toolkit for building diffusion models. Diffusion models are the technology used by DALL-E, Stable Diffusion, and other AI tools that build images through random processes.
- English is the dominant language for AI research, and that inevitably introduces bias into models. IGLUE (Image-Grounded Language Understanding Evaluation) is a benchmark that tests an AI system’s performance in 20 different languages, and includes culture-specific images.
- PromptBase is a secondary market where you can buy and sell prompts for machine learning systems. They’re currently soliciting prompts for DALL-E, Midjourney, Stable Diffusion, and GPT-3. This world is developing very quickly.
- AutoCounterspeech is a language model that generates appropriate replies that confront and contest hate speech. It’s another example of a large language that has been adapted for a specific purpose with specialized training.
- Simon Willison and Andy Baio have created a tool to explore 12 billion of the images used to train the Stable Diffusion image generator. Their results are fascinating.
- Neuromorphic computing, which is based on specialized chips that emulate human neurons, is better at identifying objects than traditional neural networks, and uses much less power.
- What does GPT-3 know about you? Possibly quite a lot; much of it may be incorrect; and some of it could be damaging (for example, being linked to “terror”).
- A teenager has built a tool that uses machine learning to detect elephants and humans in real time from infrared images taken by drones. This could be invaluable in preventing poaching.
Programming
- Stephen O’Grady’s article on bait-and-switch open source licenses is a must-read.
- Is platform engineering an alternative to both DevOps and SRE? Platform engineering is the discipline of “building toolchains and workflows that enable self-service capabilities for software engineering organizations in the cloud-native era.”
- Nbdev2 lets git and Jupyter notebooks play well together, solving a major problem for collaboration with notebooks. Collaboration and version control no longer work at cross-purposes.
- Tauri is a Rust-based framework for building desktop apps. It is conceptually similar to Electron, but uses Rust for the backend, and generates much smaller executable files.
- For those who don’t get along with IDEs, here’s a quick HowTo about running Github Copilot in the terminal with Vim. Has anyone done this with Emacs?
- Bryan Cantrill on Rust and the future of low latency embedded systems: Rust is the first language since C to live at the border between hardware and software.
- Explainshell looks up the documentation for every command and its arguments on a bash shell command line. Clever.
- HTTP QUERY is a new method that has been added to HTTP to support building APIs. QUERY requests are safe; they never alter the resource being queried. The query is placed in the payload of the request, rather than the URI. And responses from a QUERY are cacheable.
- Fuzzing is a powerful testing technique; it means watching how the software under test handles random data. Dr. Chaos is a new fuzzing framework for C, C++, and Objective-C.
- Trace-based testing is the next step forward in observability. It means using data from tests run during software development in operations, to determine exactly what kinds of events can occur and how.
Security
- Software supply chain security is more important than ever; Microsoft claims that the Lazurus cybercrime group, which is sponsored by North Korea, is adding backdoors to many widely used Open Source programs and libraries.
- Chaos is new malware that can infect both Windows and Linux devices, including routers, firewalls, and other networking hardware. It is spreading in the wild; it propagates by taking advantage of known vulnerabilities.
- Prompt injection attacks against GPT-3: Simon Willison demonstrates a new security threat that is similar to SQL injection. This will be an issue for GPT-3 applications that combine prompts from untrusted users with prompts that are generated by the application.
- The Atlantic Council has published a report describing an international strategy for securing the Internet of Things. The report is based on case studies in the US, UK, and APAC, and focuses on smart homes, networking, and telecommunications.
- Domain shadowing, in which a criminal group hijacks a DNS server to insert its own domains under the legitimate domains, without modifying the legitimate domains, is becoming an increasingly important threat.
- An experiment demonstrating the danger of automated surveillance showed that it was possible to find individuals and locations in Instagram photos using data feeds from cameras (both open and private) installed in public places.
- The popularity of browser-in-browser attacks, in which a compromised site steals information by creating a fake browser within the active browser window, is rising.
- Street View gives Google a head start on building immersive experiences of different places. Is this a down payment on the Metaverse?
- The LockBit ransomware group may be preparing to use distributed denial of service (DDOS) attacks as another form of extortion. They are also learning to defend themselves against ransomware victims who attack them with DDOS rather than paying.
Web3
- Starbucks, Chipotle, and even Universal Studios have developed NFT-based loyalty programs. Chipotle even has a simulated grilling experience, conducted in their Metaverse property.
- Cryptocurrency can be used to pay taxes in Colorado. Utah is set to follow.
- Can Web3 be used as a tool to combat climate change? Fred Wilson points to efforts like New Atlantis, for marine biodiversity, and the Toucan Protocol, a voluntary carbon market. Wilson’s thesis is that work against climate change will be crowd-funded.
- Andreessen Horowitz has introduced a “Don’t Be Evil” license for NFTs, similar (in concept) to the Creative Commons licenses. There are six distinct kinds of license, including an “exclusive commercial rights” license and a “universal license”; some licenses provide automatic revocation for hate speech.
Metaverse
- Some studies show that surgery patients who are given a virtual reality program to view during a procedure require less anaesthetic. VR may also help in post-operative recovery.
- A modeling agency is using real models to create Metaverse avatars for use in advertising. Faces are based on 3D photos; bodies are synthesized. The models are given unique voices and personalities. The avatars are sold as NFTs that expire after a given time.
- Ethereum has made the transition to Proof of Stake. PoS provides its own set of challenges, but requires much less energy and should support significantly higher transaction rates. Nothing broke, the price of the major cryptocurrencies remained stable, and the used equipment market is now flooded with GPUs.
- Neal Stephenson says that the Metaverse will “start on the wrong foot” if it leaves behind people using 2D screens. In the 1990s, he didn’t forsee the sophistication of modern gaming, specifically the ability to navigate 3D spaces with 2D hardware. Stephenson is co-founding Lamina1, a company building a “base layer” for an Open Metaverse.
- Roblox is developing avatars that can reflect their owners’ facial expressions during game play in real time.
Quantum Computing
Biology
- A memory prosthesis might be able to restore memory to people with diseases like Alzheimer’s. The prosthesis generates signals that are similar to the signals that neurons create when creating or activating memories.
- Manufacturers of high performance biomaterials, such as spider silk protein and mycelium, are starting to scale up production. Synthetic biology is becoming real.
- A new genetic therapy attempts to design human B cells, the cells that make antibodies, to target rare diseases by manufacturing missing enzymes.
Hardware
- The MAX78002 is a low power microcontroller designed for running neural networks in edge computing applications. There are claims that it can run for a week on a single AA battery. It has 64 parallel processors and can run a network with up to 3.5 million parameters.
- The Chinese are planning to build a dam with a distributed 3D printer, using no direct human labor. There’s arguably no printer at all; the work is done by AI-controlled robots that pour the concrete and roll it out in layers.
- NVidia has a new GPU chip with specialized hardware for training transformer models. It is 4.5x faster than their previous high-performance data center GPU.
- China has developed its own GPUs, the Biren 100 and Biren 104. This will greatly reduce its dependence on NVidia for high performance computing hardware.
- Battery power played an important role in helping California’s electrical grid survive September’s heat wave without outages.
