Hi all(emaal),
Good to be back! I was on a small hiatus due to a belated summer holiday and moving house, but happy to return for this year’s last AI update as little present before the year’s end.
As is customary at the end of our yearly journey around the sun we’ll be looking back at what happened this year, and try and draw out some general developments from it that we can learn from.
This is a longer newsletter than usual (understatement), with the same general topics as always but in a slightly different order. It’s become quite the wall of words, so start with the section most interesting to you I’d suggest. I hope you can find some words of worth below, and I wish you all happy holidays and a great start to the new year!
Now without even further ado-
Regulation
We’ve seen lots of action (and inaction) on the topic of regulation this year as governments and institutions scrambled to find out what they wanted to do with AI. It’s still unclear exactly what they want. If we condense the conversation down to the USA, EU, China, and global developments, we can see the following happening (disclaimer, I am not a legal expert):
USA (low regulation, high innovation): highlight was the executive order of Joe Biden as contrast to the USA’s till-then policy of ‘if companies say they’re being responsible we believe them’. Their tune seems to have become slightly more strict towards companies, but still very lenient. The vast majority of AI breakthroughs are happening in the USA.
EU (high regulation, low innovation): highlight was the acceptation of the EU AI Act, we’ll have to see what is left of it coming year for the final-final acceptance vote as the lobby from EU companies is taking a note out of the USA’s playbook and p(l)aying hard to reduce the amount of incoming regulation.. and the EU seems to increasingly listen. The strongest regulatory frameworks are still being set up in the EU.
China (medium regulation, medium innovation): highlights were the quick response regulations towards generative AI, and the careful balancing of AI innovation & regulation.. but the lack of control towards generative AI output seems to be putting increasing worries in the government’s mind who’s first priority is control of information. The most balanced approach towards regulation and innovation is (currently) being set up in China.
Global: highlights from a Western point of view were the G7 summit in Japan resulting in the Hiroshima Process - an attempt towards a global regulatory framework for AI - and the UK’s AI Safety Summit - an attempt towards a global safety framework for AI. Both agreed the events were a grand success even though nothing was delivered other than a roadmap of meetings, empty promises, and puppet institutions being set up. The (Western) world has no consensus in the slightest on how to deal with AI developments or what international regulations and safety considerations may look like, and the UN and other global organisations are being paralysed by geopolitical disorder and realpolitiks.
Health & Medicine
We have seen great potential this year come out where AI was used as a tool for discovery. From discovering new materials that are more sustainable, those that do not use animals as source, or have properties that will help tremendously in for example the transition towards sustainable energy sources. Deep Learning techniques have shown exceptional performance in discovering new drugs, like finding new antibiotics to fight the rather pressing antibiotic crisis, or predicting and generating 3D structures of proteins (the lego blocks of biology that combine to build everything from antibodies to your hair’s keratin) which dictate their properties and interactions with other proteins.
Other than discovery, we’ve also seen a glimpse of how Large Language Models and other Machine Learning techniques could help support medical practitioners in their daily work. From functioning as a second opinion for a GP or a first line of remote diagnosis to supporting in surgery, patient rehabilitation, and patient support. Although promising, the actual integration into medical practice may vary per technology mentioned due to logically strict safety precautions and a field that is not always as chippy about changing work routines (although we see that in most fields).
What is sure however is that AI is making big contributions to the field of medicine and health, and investments into AI research for this domain will likely only grow bigger as time goes on.
I am not a financial advisor.
Ethics & Security
Of course, like with any new technology, there are concerns with regard to safety and ethics, and the new advances in AI are no different. We’ve seen bad actors trying to leverage AI for illicit purposes, from using voice cloning and deep fakes to scam people out of their money by acting like distressed relatives, to the sale of a ‘FraudGPT’ bot to support in the generation of scam-messages and emails tailored to your target demographic. We’ve also seen the effects of the improper usage of the technology leading to safety issues, like when an employee of Samsung accidentally leaked company secrets on the public version of ChatGPT, or when people input private information (like a Dutch BSN number) into a prompt.
Furthermore, we’ve already seen research come out that the use of a LLM subtly changes the user’s thoughts and opinions, which combined with the fact that companies dictate in what way a chatbot answers questions leaves us with quite some big question marks (and exclamation points for that matter). Lastly, we have seen proof that LLMs are intrinsically unsafe, and as the safety issues are a structural problem with the Machine Learning technique itself (transformers) it is unlikely to be solved - something to keep in mind.
Looking ahead, I think it’s likely we will see a situation akin to social media; quick adoption due to a race for profit, with negative side effects only truly becoming clear once it’s already spread everywhere - afterwards leaving us with the question whether the positives were even worth the negatives.
Slow and steady guys, slow and steady.
Business
It has become clear that the recent AI advances showcase enormous potential for productivity increases, yet, only when used properly. A job is a collection of tasks, and many tasks can increasingly be automated or augmented through proper use of AI software - but the integration of these new technologies should always be done in open and honest conversation with the employees whose work it will change. This productivity gain also means that companies will increasingly feel the pressure to integrate various AI technologies to keep up with the competitive frontier, potentially making them mandatory to stay afloat.
Data is increasingly becoming even more valuable for companies to keep, as it becomes easier and more accessible to train (or finetune) AI models for the specific purposes and customer base of the business. Pair this with the public internet increasingly becoming a polluted pool of synthetic (i.e., generated) data, and there’s an increasing incentive to have a proper data architecture and collection set up and ready to take advantage of the capabilities of current and future AI models.
We also see many big tech companies investing into their own production of specialised chips, and even further into their own full (AI) chip manufacturing supply chain. This trend seems likely, to me, to continue, especially with the global semi-decoupling of supply chains due to geopolitical tensions and the great strategic value given to AI tech and development.
Throughout all these great investments and developments, there’s a striking absence in clear pathways of big foundation model development (like OpenAI’s GPT4, Meta’s LLaMA2, Google’s Gemini) towards monetisation and recouping of costs. The current scheme seems infeasible in the long run, with an evolution of today’s situation both inevitable and hard to predict. Biggest bets seem to be on enterprise software (like finetuning GPT4 for a company’s own database), and creating a new ecosystem (like how OpenAI wants to create a marketplace for ‘GPT’ bots).
Whatever happens, you can be decently sure the future offerings won’t be as pleasantly offered as today’s, so make sure you stay agile and don’t make the LLM or Foundation Model you use a single point of failure, you want to be able to switch them out easily when services change.
Ah, and always remember, simple solutions can often work almost as good as fancy new solutions at a fraction of the cost, and always benchmark the new situation compared to the old situation! (And use both quantitative and qualitative data for performance reviews!!)
Creativity & Art
Artists and creative professionals have had quite the rollercoaster ride this year. The impact of new generative AI tools and techniques have caused both enormous concerns and great opportunity, seemingly with little middle ground between the two extremes. We’ve seen artists utilising the new technologies like Large Language Models to give a voice to the climate, to artists seeing their work being used without their consent for the training of an AI model that threatens their financial future. In some creative industries, generative AI can help creative professionals tremendously by minimising grunt work, like in the games industry by helping small team (indie) developers create games for less, or by supporting creative professionals through creative hurdles, like by helping writers overcome their writer’s block. These augmentations also show potential, however, to remove parts of one’s work that one truly treasures in their creative process, or to automate so much of one’s original work that their job comes under threat.
Of course, copyrights were one of the big issues this year. Training data for large AI models are opaque, it’s the secret sauce the AI chefs use to create their highly prized products - yet there’s little oversight whether the ingredients in the sauce are ethically sourced and not potentially dangerous. Just like a secret tomato sauce recipe for an incredible pasta dish (hint: oftentimes it’s just loads of butter), it becomes very difficult to discern what exact ingredients are in there once the results are on the plate in front of you. There’s some ways to, more or less, find out if certain ingredients are in there, like an AI model understanding how to create a Picasso’esque image, but for more niche artists this becomes almost impossible.
I think in the field of arts and creativity, there’s both enormous potential in the new technologies coming out, and tremendous harm hiding behind it. It seems to me that art’s peculiar position within a capitalistic system, which rewards easily discernible lines towards monetary value creation, is coming ever more under threat. On the other hand, creative professionals that are able to make the technology work for them seem well set-up to profit - as of now AI does not have the aesthetic eye, ability to deal with constraints, and capability to create entire new paradigms in art like creative professionals have. Non-commercial artistry and personal creative practise will similarly not be too negatively affected, and potentially even positively impacted, by these developments (using my very limited and very biased perspective).
Important footnotes here is the increasingly pressing question of what we appreciate in art, what it even is, and what can (and cannot) be creative. This leads into a larger discussion on whether humans should have the right to know whether something is AI generated or not, of which I am a proponent.
Now just to figure our when something is AI generated or merely AI supported.
Education
Take a breath educators, it has been quite the year. The advent of Large Language Models (LLM) that easily generate texts, ranging from essays to peer feedback, have created quite the headache due to its quick adaption by students. Short term, the question of test validity and educational governance was put forward - most seemed to land on verbal testing as a decent compromise, and to teach for proper use rather than sweeping bans. Conversations though are anything but done on the topic. Long term, the question of what world we educate the new generation for, and what that means for education practises is becoming more pressing. And, if I may add, the question on a broader level how to adapt education structures such that it becomes more agile in responding to the structural disruption that technological developments bring forward.
We have seen educational institutes embrace new AI technologies, by using them to lower administrative clout, or as a personalised tutor away from school. We have also seen sweeping bans and attempts to gain control over the technology, with an emphasis on finding methods to recognise AI-generated content. Sadly for this last group, classification of AI-generated textual content is insanely difficult, way too easy to circumvent, and has too high a false positive rate (e.g., classifying a student to be fraudulent whilst they aren’t) to be ethically put into practise.
New technologies, now LLMs and chatbots, but just like smartphones and social media before them, are leaving their marks behind on society in an ever faster pace. One of education’s primary duties is to prepare the next generation for the future that is yet to come, to teach them the skills and knowledge necessary to navigate the world that will be. That duty is becoming ever more difficult, as our sight of the future becomes ever more obstructed by thick fog and twisty and winding paths. If we are to educate for the future, we need to tread carefully but steadfast, to set steps forward but not be afraid to change direction once we find out the wrong path is being followed, to experiment with different directions and learn from each other. To find out what we need to know, what we need to be able to do, to not just survive but thrive in a world who’s one constant is change. What we just can’t do is stand still, and risk the next generation to set their first steps into the world unprepared for the encroaching fog.
Models & Datasets
This year has seen an absolute avalanche of models, from tiny to gigantic, from super specialised to as general as possible. For a comprehensive summary of all developments I’ll point you to the State of AI Report 2023, but to summarise in a single sentence i’d say ‘built big and wide to go small and narrow’ more or less captures this year’s spirit. We’ve seen model sizes balloon to humongous proportions to capture generalistic learning, only to be shrunk down and finetuned to specific tasks. This dichotomy seems to continue unto the next year, with transformer architectures encouraging the training of models over as-large-as-possible datasets to gain access to ever more ‘emerging’ abilities.
There are some challenges on the horizon, however, for those wanting ever more data in their models. Firstly, the amount of public data available for training (disregarding stingy topics like ‘ethics’ and ‘copyright’) is quickly becoming too little in size. Secondly, the public data available is becoming more ‘contaminated’ with synthetically generated output by the day. As ever with the data quality vs quantity question, a lack of quantity can be made up for with higher quality data, something that increasingly is on the forefront of model design (echoing developments over the last years towards data-centric ML). Larger focus is also falling towards synthetic data creation to play a large part into training data collection as a solution to the above, although it is not a perfect alternative.
Looking forward, I seem to spy some of the following trajectories. Models, it seems, are integrating ever more modalities, with embodiment into robotics being a next frontier with great potential impact. Next up on the line of Foundation Models are Large Vision Models (although public images are less functional for enterprise use as compared to text). More (actual) open source models and datasets will be coming out, with robotic (simulation) data seemingly next in line for the open curation of big datasets. Integrating logic/causality and effective constraints - and cohesive ‘storyline’ generation for non-textual output - seem to be big focal points for improving upon existing models and architectures.
But like all developments in AI nowadays, there could be a paper tomorrow that puts everything on its head. Hope everyone likes rollercoasters, we’re well stuck in one.
Etc
I recently became aware of the Royal Institution’s Christmas Lectures, where fittingly this year the topic was AI. If you’re so inclined, it might be fun to peruse either this year’s or previous years’ lectures.
In truly disastrous news, it seems many Dutch folk mistakenly pronounce ‘gènerative AI’ as ‘genèrative AI’ - okay this is just my personal pet peeve but I hope to do my best effort here to get the proper pronunciation into the Dutch zeitgeist. ‘JEH’nerative, not ge’NEH’rative!
In more disastrous linguistical news, the term ‘Artificial Intelligence’ is continuing to deform into meaning ‘anything to do with computers that do slightly smart stuff’. To be fair, this is also on us AI researchers not being able to come to a consensus what exactly AI constitutes in the first place. Sorry!
And lastly, a bit late, but if you enjoy excruciatingly hard riddles, the New Scientist has a list of 15 science-related ones that will keep your brain juices from going stale over the holidays.
See you in the next year, be safe with the fireworks if you’re in The Netherlands, and have a good one!
With the warmest of regards from the safe and quiet UK countryside.
Leven is mooi