It’s impossible to keep up with all the new developments in the LLM-era. However, one thing has been true: they never stopped improving.
Malte Skarupke explains How LLMs Keep on Getting Better, covering a few of the different visible and invisible aspects of LLMs that have been worked on over the past couple of years. It’s a really good overview for those who are not into the weeds of it.
(via Antónia)
An excellent layman’s recap on the dependency (in terms of defense, but also economy) that Europe has on the US tech. What happens if we cannot have US-owned operating systems in our mobile phones? Or we cannot buy American brands for our hospital computers and servers? Will you still receive emails or direct messages?
I will continue my quest to move out of gmail to something European. Unfortunately, Portuguese SAPO is no longer an alternative, so I will have to go for something German, Dutch or Swiss.
I use Hacker News as a tech and economy news feed, but I don’t necessarily comment or upvote a lot.
Just like Youtube or Spotify’s wrapped (I personally use Last.FM), there is this fun HN wrapped that was done with a tongue-in-cheek style that I particularly love.
Conceptually, Mastodon is a bunch of copies of the same webapp emailing each other. There is no realtime global aggregation across the network so it can only offer a fragmented user experience. While some people might like it, it can’t directly compete with closed social products because it doesn’t have a full view of the network like they do.
The goal of atproto is enable real competition with closed social products for a broader set of products (e.g. Tangled is like GitHub on atproto, Leaflet is like Medium on atproto, and so on). Because it enables global aggregation, every atproto app has a consistent state of the world. There’s no notion of “being on a different instance” and only seeing half the replies, or half the like counts, or other fragmentation artifacts as you have in Mastodon.
I don’t think they’re really comparable in scope, ambition, or performance characteristics.
— Dan Abramov, Hacker News comment discussing his Open Social article (via Simon Willison)
I did not know about this huge difference. The reason why federated social networks will never work is because it is done at the cost of usability. Discovery and recommendation is done better when everything is centralized. atproto gives everyone a global view, which has scalability issues (if you care about recommendations for instance). FAANGs have been tackling these scalability issues for the last couple of decades in the centralized (but physically distributed) environment. I doubt it can be done completely federated.
On the other hand, I believe we need more players in their fields (Netflix is one of the few that does not have a monopoly), but other major tech companies are in one or at least a duopoly. Federation is a good solution to breaking those monopolies.
But I do not see that happening, unfortunately. The customers in this day and age care about low friction more than privacy or price (especially if you can pay with time or attention).
Om Malik interviews iRobot founder Rodney Brooks:
At MIT, I taught big classes with lots of students, so maybe that helped. I came here in an Uber this morning and asked the guy what street we were on. He had no clue. He said, “I just follow it.” (‘It’ being the GPS—Ed.) And that’s the issue—there’s human intervention, but people can’t figure out how to help when things go wrong.
Taxi drivers used to have to know every single street in the city to get their license issued. TVDE don’t even need to know street names. If you ask for the directions to a Portuguese-named hotel in Lisbon, they ask you to type it in their phones. Navigation apps have done a disservice in now being designed to teach POIs and navigation to humans. Let’s hope you have a power bank near you when you get lost in your own city!
We’re trying to put technology in the manual warehouses, whether it’s DHL—our biggest customer—or Amazon. It’s about putting robots in places where there are no robots. And it’s not saying it’s a humanoid that’s going to do everything.
You’re right, it’s not sexy. And you know what that means for me? It’s hard to raise money. “Why aren’t you doing something sexy?” the VCs ask. But this is a $4 trillion market that will be there for decades.
Software Companies (Microsoft, Google, Facebook) have shifted the mind of VCs. Due to how fast it spreads, it was much easier to obtain monopolies (and make a lot ton of money) than with previous life-changing inventions (phones, computers, cars). Everyone is looking for the next unicorn like it’s the Gold Rush.
I always say about a physical robot, the physical appearance makes a promise about what it can do. The Roomba was this little disc on the floor. It didn’t promise much—you saw it and thought, that’s not going to clean the windows. But you can imagine it cleaning the floor. But the human form sort of promises it can do anything a human can. And that’s why it’s so attractive to people—it’s selling a promise that is amazing.
This is what you get when you study interaction design. Physical affordances and skeuomorphisms. If you were a 18 century time-traveler you would be more likely to be able to use an early iPhone than the current Liquid Design ones.
I think we need multiple education approaches and not put everything in the same bucket. I see this in Australia—”What’s your bachelor’s degree?” “I’m doing a bachelor’s degree in tourism management.” That’s not an intellectual pursuit, that’s job training, and we should make that distinction. The German system has had this for a long time—job training being a very big part of their education, but it’s not the same as their elite universities.
In Portugal, the technical schools and universities are now offering the same courses (given in the same style), including PhDs, with no distinction. Diversity is healthy and should address the dichotomy of learning to get a job, and learning to change the world. Both need distinct methods and depths.
As 3D printing becomes more general, in the same way information technology and payment systems got adopted in the third world more quickly than in the US, 3D printing will become the engine of manufacturing.
Right now, the supply chain is the reason China is so effective. Chinese manufacturing companies realized they had to diversify and started building supply chains in places like Malaysia, Vietnam. But if 3D printing really gets to be effective, the supply chain becomes all about raw materials that get poured into the front of those 3D printers. It’ll be about certain chemicals, about raw materials, because then every item would ultimately be 3D printed. That completely breaks the dynamic of what made Chinese manufacturing so strong—the supply chain of components.
Brooks is the first person to provide me with an optimistic viewpoint of manufacturing and China’s upcoming world dominance.
Well worth the read!
..
As pessoas em Portugal votam como fãs de partidos como se de clubes de futebol se tratasse. Parabéns à equipa do Lisboa Para Pessoas pelo trabalho nesta plataforma que agrega os programas dos vários partidos para os 18 municípios da zona de Lisboa.
Um sampling nada aleatório:
Nova Direita para Lisboa: Implementar mecanismos de dissuasão da criação de bairros de hegemonia religiosa.
Espero que não se esqueçam da religião católica. Vou adorar o baralhar e dar de novo que vai acontecer com a habitação em Lisboa!
CDU para Lisboa: Defender junto do Governo e da AR que os municípios devem poder, perante o reconhecimento da declaração de situação de carência habitacional, proceder à posse administrativa de fogos com uso habitacional, devolutos ou sem utilização há mais de um ano, após a notificação.
Uma pessoa faz uma sabática e fica sem casa. Que giro!
“Moedas para Lisboa:” 100% da frota da Carris com zero emissões até 2030.
Alguém que me explique como é possível ter zero emissões num transporte público. A matemática não faz sentido.
Reddit says that it has caught AI companies scraping its data from the Internet Archive’s Wayback Machine, so it’s going to start blocking the Internet Archive from indexing the vast majority of Reddit. The Wayback Machine will no longer be able to crawl post detail pages, comments, or profiles; instead, it will only be able to index the Reddit.com homepage, which effectively means Internet Archive will only be able to archive insights into which news headlines and posts were most popular on a given day.
— Jay Peters for The Verge (via Simon Willison)
Next it will be google. And most of the post-2025 forum knowledge in the web will be lost. Imagine a web browser that cannot access StackOverflow or Reddit. How useful it is? LLMs will need new data to continue being relevant, and a new data monetization strategy will change the internet forever.
 |
 |
 |
IconFactory is a boutique design studio that focus on app icons. Their business is being replaced by AI.
This is the challenge of GenerativeAI: it replaces creative work (or automates it). Design work is one of those fields where the number of employed designers will decrease, and you will need to be either a very good, or a very productive one. If I were an undergrad in design, I would try to learn the difficult stuff, and now the run-of-the-mill stuff that will be easily automated.
Places like Hong Kong and Tokyo have a lot of commuters, leading to a lot of congestion around station gates. Sony realised this, and invested heavily into the performance of their technology – FeliCa cards boast an advertised communication speed of up to 424kbps, making a noticeable improvement in gate processing speeds compared to Western counterparts. Compare the speed of passing through a ticket gate on the Underground to a Tokyo ticket gate – you could practically sprint through. This is partly achieved by the fact that transactions only involve the card and the reader itself – the reader doesn’t talk to an external server to perform a transaction. This makes IC cards stored-value cards – as in, they store the value on themselves, rather than their value being stored on the backend where it’s controlled fully by the operator.
— Japan’s IC cards are weird and wonderful by @aecsocket
I visited Japan in May and I was a bit confused by how IC cards (pre-paid NFC cards) interacted with my iPhone. It was really weird that I could have a digital version of the card or a physical card, not both. In practice, when I converted the physical card to Apple Pay, the physical copy would no longer work. After reading this awesome article about the technology, I now understand why: unlike western NFC cards, the money is stored as credits in the card itself. Therefore, you are limited to having only one of them as your money storage device. I wonder whether IC cards could be used for money laundering, given how multipurpose they are — you can pay your supermarket or mean with them!
Apple Pay is very convenient. Most of the days I don’t even carry a wallet. I pay everything with my card, and I even have my citizen ID and driver’s license on my government app. However, I do not carry a power bank or a lightning cable. That means that I’m usually screwed up if I ran out of battery. In Japan, that meant getting stuck in transit (especially when traveling from city to city). The fact that NFCs can work passively is a major advantage of the tech. Maybe we need phone NFCs to work even without battery. Or have some kind of chi-charging and a low-battery mode for phones to provide critical features outside of the main OS.
Recommended audience: Startups and large companies who intend to sell software to universities.
Most SAAS is sold on a per-seat basis. But this does not scale to universities, as we have a large number of possible seats, but most of them (students, possibly from different scientific areas) do not use the software, at least for it to be worthwhile.
On the other hand, unpredictable costs (when paying per activity) is also something that does not work, as we need other budget it yearly.
Chris Siebenmann has a really good write up on this issue, which I recommend if you manage or sell to universities.
I’ve never stop to consider that footnotes don’t make sense on the web. I still see their point, but maybe there should be an html element for that, with browsers using a tooltip style on-hover popup.
Stuart Langridge makes good points about keeping using footnotes.
If you have no in-house authentication or ‘who exists’ identity system and you’ve offloaded all of these to some external provider (or several external providers that you keep in sync somehow), you’re clearly at the mercy of that cloud identity provider. Otherwise, it’s less clear and a lot more situational as to when you could be said to be using a cloud identity provider and thus how exposed you are. I think one useful line to look at is to ask whether a particular identity provider is used by third party services or if it’s only used to for that provider’s own services. Or to put it in concrete terms, as an example, do you use Github identities only as part of using Github, or do you authenticate other things through your Github identities?
I was a very strong advocate for OpenID and OAuth, back in the day. However, my idea was to own my online authentication. My OpenID was alcidesfonseca.com, which I would delegate to a provider of my own choosing, which I could change whenever I wanted, without changing anything on the several websites I used to authenticate with.
However, the silo’ed web decided against custom OpenID providers, and ended up supporting Google, Facebook and Github, for the more tech savvy websites. And I’m talking about important stuff, like Tailscale (my VPN provider). The open web lost its decentralization.
Now that I am moving away from Google, I have to change my login details in every website that I used Google as my identity. But most do not support adding a tradition email-password login. And I don’t quite want that. Because I don’t want to update passwords every once in a while at a 1000 websites.
So remember kids, create an account on every website, so you can leave your centralized server whenever they change their privacy policy or they start charging for what used to be free.
To assess the productivity gained by LLMs, Iris Meredith distinguishes between transformative and compositional work:
While work is obviously a very, very complicated thing, a useful lens for the purpose of this essay is to draw a distinction between work which reshapes a raw material into a finished object and work that puts together multiple objects in a way that creates a certain effect in the world. For the sake of having a shorthand, I’ve chosen to call them transformative and compositional work, respectively.
While Iris does not have a particular clear idea of LLMs, Jeremy Keith does:
My own take on this is that transformative work is often the drudge work—take this data dump and convert it to some other format; take this mock-up and make a disposable prototype. I want my tools to help me with that.
But compositional work that relies on judgement, taste, and choice? Not only would I not use a large language model for that, it’s exactly the kind of work that I don’t want to automate away.
Transformative work is done with broad brushstrokes. Compositional work is done with a scalpel.
Personally, I think it depends much more on where you are in the economic value proposition. Are you selling quick-and-dirty cheap stuff? LLMs are great for you. Are you delivering high-assurance, high-quality work? Then you might/should be skeptical of LLMs.
Gorillaz’s Damon Albarn and Kate Bush are among 1000 artists who launched a silent album (on Spotify no less) in protest against the UK government allowing AIs to be trained using copyright-protected work without permission.
This protest highlights the tension between creating valuable tools and devaluing human content.
ChatGPT (and even Apple Intelligence) is trained on information publicly available on the internet, data from (consenting) third parties, and information provided by their employees or contractors. Over the last year and a half, people have been amazed at what ChatGPT has been able to do. Although the quality of its work fluctuates as new data/methods are being updated, ChatGPT and similar tools are being used to create value. But at what cost?
Unconsciously, The Algorithm has become more and more important in our lives. From Instagram and TikTok reels, X and Facebook timelines, Spotify, YouTube, or Netflix’s recommendations, the decision of what we see is no longer ours. And we are also not delegating our choices to a human editor (as is the case of the old boring telly or radio channels). Those decisions are being made by black-box algorithms that are hidden in the shadows.
The EU AI law, which I blogged about before, only requires explainability for applications in high-risk domains. Entertainment can hardly be thought of as high-risk. However, I would argue that given the importance of online content consumption in today’s society, it should be considered high-risk. One example is the perceived power of Twitter/X in political elections.
On the other hand, educational purposes are considered fair use in most countries (which is certainly true here in Portugal). What is the difference between fair use for human and machine learning? As we become increasingly dependent on AI for our daily tasks – I use Siri and Reminders to augment my memory and recalling ability — we become de facto cyborgs. Is there a difference between human and machine learning for education?
In 2017, Spotify introduced the Perfect Fit Content program, encouraging editors to include songs purposely designed to fit a given mood in their playlists. Liz Pelly goes into all the details in her piece The Ghosts in the Machine. Some human, some AI, several companies have been starting to produce music à lá carte for Spotify.
According to The Dark Side of Spotify, Spotify investors are also investing in these companies (phantom artists on the platform, which use random names with no online presence other than inside the platforms) and promoting the use of AI to beat the algorithm. While this vertical integration might be cause for considering anti-trust or monopoly issues, the fact is that Netflix has been successful in expanding to content production (as has Disney been successful in expanding into content distribution).
AIs are much more productive in generating music than humans. Which is not necessarily the same as being successful in producing music a) that humans enjoy or b) that is commercially viable. The Musical Turing Test is almost solved, addressing a). Commercial viability is even easier to address. Because the cost of producing AI music is so low compared to the human equivalent, AI companies can flood the market with millions of songs, letting the algorithm filter out the ones that do not work. In that scenario, human musicians are not just competing with each other for user’s attention but are now unable to be showcased to users without an explicit search. Additionally, AI can better cater to some audiences based on data extracted from these networks (remember Spotify’s investors also investing in AI music production companies?) than humans can, at least in large numbers.
And I’m aware AI can be a tool for musicians, but if AI can perform end-to-end music generation passing the Musical Turing Test, it becomes much more interesting from a commercial standpoint.
The only chance for musicians is to promote their own content outside of these platforms, abandoning the initial goal of Web 2.0, where anyone can create content on the web. They can, but it just won’t be discoverable in the ocean of AI-generated content. But this is a symptom of a more significant problem for the web.
I feel like the people who try to be positive – well, I don’t know what they’re doing. I’m a music producer and also a writer who also happens to make art/design her own album art. Thankfully, I also dance, which is going to be the one thing that saves me I feel. — PrettyLittleHateMaschine on AI music.
ChatGPT was primarily trained on internet-available content. So, its quality depends on what is available at a given time. If we stop collecting new information, we can assume its quality will remain unchanged. Still, it will not be helpful with new information, such as news updates or scientific discoveries. Its usefulness will be reduced.
On the other hand, if the quality of AIs increases — it’s more and more difficult to tell the difference between human and GPT-generated text — and it passes the Turing test, the content available online will be more and more AI-generated than human-generated, as it’s more economical to use AI to produce text, audio or even video.
Here, we consider what may happen to GPT-{n} once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear.
— AI models collapse when trained on recursively generated data
This recent Nature paper reports that LLMs perform worse when trained on LLM-generated content. Human content is now essential! LLM companies need high-quality human content to train their next-generation models, especially concerning novel knowledge. But econmics no longer work. Content is created once, consumed once, and used to generate millions of derivates for almost free. An author might publish a book, hoping to make the money for the time it took to write from the sum of all individual sales. However, AI companies will not buy the book at its production cost to train a model. Same for daily news. The human audience is still needed to make this work. And suppose everything is made available for free on the web. In that case, humans are making the same mistake that led to ChatGPT being in business without contributing to the original content sources.
Web 2.0 died and now the web happens more and more inside silos. Famously, Instragram does not allow for links outside its app. “Link in the bio” will be listed as the cause of death in Tim Berners Lee’s obituary. It goes against what the web was supposed to be. But today’s personal entertainment happens in silos (Instagram, Netflix, Disney+, etc…), not on the open web. Even Reddit communities have started blocking links to some websites, like X.
The web failed at microtransactions. Paying 10 cents for reading a well-written article was the original goal. Even with Paypal and Apple Pay, the model was only successful for large purchases, not pay-per-view. Imagine that you give Youtube your credit card, and it takes 1 euro for each hour watched. Once you have something for free, it is difficult for companies to make you pay for it.
As a business that moved from analog to digital almost completely, most news outlets have failed to change their economics and they are now struggeling financially. As the price of advertising online has decreased over the past years, they have switched to a subscription model, putting up paywalls with dubious outcomes.
I foresee a web where high-quality human content is behind paywalls. While most of the web can be AI-generated and free, it will be ignored if high-quality content is available from trusted sources. Content will be signed and (possibly) encrypted using personal keys. These keys can be provided by the government, or other parties. For instance, every Portuguese citizen already has their keys inside our citizen cards, sometimes with professional attributes.
If you wanted to read the news, you can go to an online newspaper, where the content will be signed by a recognized journalist or editor. The body of the text can be encrypted but with a faster Apple Pay-like prompt, you can pay cents to read it. Even if the journalist published AI-generated content, they are liable for its content.
This proposal makes the web a more trustful place and somewhat addresses the economic problems of paying for content on the web. It requires payment processors to drop the minimum cost per transaction, which I believe is happening more and more. And as more and more garbage is published online, users will see the need to pay for high-quality content.
As for AI providers, they will now have to pay for content. And even if it is ridiculously cheap, there is a trace that they bought that information, useful when you want to prove in court that your content was used in training LLMs.
We might not get to this Web, but I hope some of this ideas help the web survide the tsunami of garbage content that is starting to flood our dear World Wide Web.
TikTok was banned in the US The Congress and the previous president banned TikTok in the US with effect on January 19th unless its owner, Chinese ByteDance, sells the app to a non-Chinese company. The current president gave it a few more weeks, but it’s still an ongoing issue.
The reason for the ban is homeland security: Chinese entities can use TikTok usage data and content to gain access to private data on US citizens, as well as use personal content for blackmailing and influencing the public perspective. They do have a point about security, but should we also ban Facebook, Instagram, Google and others for their possible control of Europeans? We need to fight for our technological independence.
However, I would agree with the ban if the reason was Public Health. I see the effect TikTok and other short-form media (Reels and Shorts) taking over media consumption, even in my age group. This results in people (even more pronounced in children) having less patience and unable to concentrate for a long time without some form of stimuli.
I don’t mean to sound like an old man, but this is becoming an issue of Public Health. And if you don’t want your government to shut down “addictive apps”, we must self-regulate its use first.
A few decades ago, people used to walk bare naked. If you were a little better off, you’d maybe wear them to church, but not on a daily basis. In 1928, Portugal forbid walking barefoot in its two main cities. The cost of shoes, even for poor people, would be nothing compared to the cost of losing a foot or a leg from an untreated infection.
Think of the case, which is quite common, of a humble head of a family who injured his foot with a piece of glass, a stone, a tack, or in some other way, and who, as a result, contracted an infection that resulted in his death. Evaluate how that family will live, whose only support has disappeared thanks to lack of foresight, which is the result of a bad habit. (Government ad to promote the use of shoes, translated from an old style of Portuguese)
More recently, the Covid curfews implemented in many countries were a similar measure: taking away people’s freedom allowed the health of the population to improve. And these are not democratic decisions. They are mostly based on a technical analysis.
Now, there are technical reports that TikTok is affecting Public Health. It is the first time the new generation has more difficulties learning than the previous one. Mental health is at its worst and we need to act upon it.
I don’t have TikTok installed. I don’t watch reels or shorts. I try to watch and read long-form content. I teach my students that you need a larger-than-average attention span as a programmer. You should train it. And we should encourage the next generation to appreciate long-form.
I’ve been following Gwern.net for a long time, and I was really curious when I found out this interview that kept him anonymous.
How does an hearing-impaired, introvert American grow up to be a polymath and a scholar?
The wikipedia editor, now full-time writer without a salary
got a lot farther by writing a lot and harder,
by being a lot smarter, by going down rabbitrabbitholesholes
by fourteen, they placed him in charge of wikipedia edits
🎶
And it’s a bold move on LG’s part, considering most folks just want to see their family photos or some calming art while they’re not actively binge-watching. Even if you can turn it off, the default setting is a bit of a slap in the face for anyone who thought they’d bought a premium product free from such annoyances.This shift towards monetizing every idle moment on your TV is a slippery slope. It’s not just about selling hardware anymore; it’s about squeezing every last cent from customers, and brokering the data to get more revenue. And while LG claims this will boost brand awareness, one has to wonder if viewers will just tune out entirely (or worse, switch to a platform that respects their downtime). As the lines blur between content and advertising, it feels like we’re all just one step closer to a world where even our screen savers are working overtime.
— LG adds Ads to Screensavers by Rui Carmo
Someone should send a copy of Black Mirror’s Fifteen Million Merits down to LG leadership. As long as profit is all these corporations care about, and as long as the users don’t stand up to this, it’s all going downhill from here.
Because I’ve always seen reading time as such an incredibly personal thing, I’ve never once paid any heed to these widgets. In fact, I’ve been slightly perturbed a service would presume to know how quickly I could read an article.I’ve always viewed any service that sticks a “reading time” widget on its articles as the literary equivalent of fast-food: you’re not here for quality, but for expediency.
Personally, I think they devalue a text more than they add to it.
Estimated Reading Time Widgets — Jim Nielsen
I’m with Nielsen on this one. It makes no sense to suggest an (average) estimated reading time, unless you are trying to convince your audience that this is not a long piece, and you can read it in a minute or two. For personal blogs like this one, I really don’t care if people read it or not. But if you are writing one of those linked-in-self-promotion-bullshit-crypto-AI-latest-fad types of posts, then your goal is to get the reader to take its time. And in those cases, you can bluntly lie and always say it’s one minute.
My guess is that this trend comes in response to instagram/reels/shorts/tiktoks where people do not want to spend any more time on a particular content than they need. Long-form reading is dying, especially on the internet.
Other than teaching how debilitating this is to the younger generation, I hold a very pessimistic view for the next generation.
