Schneier on Security

Clive Robinson • June 20, 2024 1:51 PM

@ Bruce, ALL,

The problem with killing the squid page is that a very significant number of news stories that later became threads were fed into the blog.

This “pre-selection” gave everyone the chance to read the story in advance of it becoming a thread and could thus get their thoughts in order etc.

Now a story that is very old to me and I’ve been waving a red flag over which is the death of E2EE on single devices is now here for all commercial consumer OS’s from Apple, Google, Microsoft because of the now AI hardware in the CPU “neural network support” systems. All of which now will sit there and read the User Interface for “Surveillance Purposes”

The Bull-Scat put out by Apple and Microsoft is “Child Protection” it’s nothing of the sort. It monitors everything you do and in MS’s case updates a series of databases every 5secs or if the database is online in the cloud as often as it can do.

The story the other day that MS were pulling back on it is “a steaming load” as they say as it’s a Beta product in Win-11 that is still very definitely moving forward.

Any way although I’ve been warning about this killing of E2EE by end-running it on the device, and giving advice and mitigation for over a decade… For many it’s snuck up on them apparently without warning, which is why videos like,

https://m.youtube.com/watch?v=c52pKpYeZ74

Are starting to appear.

Clive Robinson • June 20, 2024 7:54 PM

@ Charles

“You have misunderstood my intent, along with the proposed implementation.”

Err not really I’ve “previous knowledge” of how such systems work in the real world rather than in theory, and the two are more than a country mile apart for “legal reasons”.

“If you have ever been to a nightclub, you will know that many of them do identity and age verification at the door, and will give you a wristband or stamp to indicate that you have passed the verification.”

By law in most places they are not allowed to record your ID, but such laws do not apply “online”, in fact the very opposite for a multitude of reasons, including US companies claiming they were “safe harbour compliant” when they were nothing of the sort. Hence the increasingly more stringent EU legislation.

But I have already described such a physical world “receipt” system. And the reason the stamp is not traceable backwards is that it is a physical real world tangible object that is too complex to use “serial numbers” or similar to make it traceable with (although time-stamp and CCTV systems are now being combined for “Patron Safety”).

Online systems however, supposedly to “prevent fraud”, are almost always a 100% traceable system with the equivalent of crypto protected serial numbers.

All this US company noise about anonymous tokens is nothing of the sort. Because they keep a record of both sides of the token. As either a user or business you only ever get to see one side. The token arbitrator how ever gets to not just see but record both sides. This they can and do store away “to be compliant with US Legislation” and also “give for free” to the US Government for “legal immunity” protection.

Clive Robinson • June 22, 2024 4:08 PM

@ Bruce, ALL,

Article on historic audio encipherment (SIGSALY) from WWII and how some of the basic parts are still in use in the entertainment industry.

https://thequietus.com/culture/books/sigsaly-cryptography-turntables-muzak-radio-phonics-laboratory-justin-patrick-moore-extract/

Clive Robinson • July 3, 2024 6:07 AM

@ Winter,

Re : Digital v. Biological

“Biological neural networks are “based on statistically weighted spectrums that are multidimensional and applied filter masks”.”

Actually we are very far from sure on that, even in the general sense.

We see some “structure” but “and insufficient detail”. Whilst “Digital Neural Networks”(DNN) are argued to be based on “Biological Neural Networks”(BNN) that is a little like arguing a cardboard box is designed on a mountain cave.

The design of a DNN neuron is a linear “Multiply and ADD”(MAD) structure which whilst linear for each multiply, has way to many inputs for the ADd to avoid various underlying issues (insufficient bit width being one). The output function of the DNN is where the summation is “rectified” which again has significant issues.

The way a BNN appears to work is by an interesting form of quite complex and nonlinear integration in the time domain. Importantly the nonlinearity is at the inputs prior to integration not the output as with the DNN.

So apart from the superficiality of multiple inputs and multiple outputs DNNs are not at all like BNNs as has been described in the literature.

It’s a case of “squinting at distant objects and hoping they are not mirages” currently.

Clive Robinson • July 3, 2024 10:28 AM

@ JonKnowsNothing, Winter, ALL

Re : Is AI going to scale to AGI

There us much debate about AI and AGI and the latter is getting more hype than the Emperor’s new wardrobe.

The fact is AI scaling is already hitting one of several walls and my personal thoughts are,

“The original AGI definition is never going to happen the way we are currently going about things.”

Some would say harsh, and I suspect some schills in the bubble pumping game will scream aloud and wave their arms and have not just fits of
conniption but make threats in various forms.

But the thing is those pumping the AI bubble, not just over stepped the mark, they leapt blindly into the dark void of daydreaming. From the fact they’ve been reeling things back in rather dramatically and quickly, and that saying they are “watering down the previous projections” does not do their back peddling justice.

I’m not the only person to notice this,

https://www.dwarkeshpatel.com/p/will-scaling-work

But the question few ask is about “compression ratios”.

That is as you compress information at some point you cross a threshold where the compression becomes lossy. The thing is what do you loose, how, and how does it show up.

Information can be compressed into a number, or the difference between two numbers but not uniformly.

Take a standard scale of numbers, all the information in the universe could be represented by just a single number or ratio between two numbers. All you need as the universe is believed to be finite is numbers of near infinite resolution that “exist but don’t exist” that is they exist in theory but not in practice.

Now consider that the closer your compressed number is to any other number that is finitely expressible the more accurate it is. So you can develop an “error function”.

But what effect does that error have?

Also what scope does it have?

And so on.

The simple fact is we can make a very simple calculation which is to count all the “significant” bits in the multiplier constants –weights– in the DNN first layer and that will give you the upper limit.

So the smaller the DNN the higher the error rate.

Thus how small can the DNN of any domain specific LLM be?..

Clive Robinson • July 4, 2024 10:22 PM

@ Winter

Re : Neurons, DNNs and no synapses.

As has been noted neurons have been studied in quite some detail and are highly complex in nature. Attempts have been made to reduce them down to mathematical models of which the work of Hodgkin and Huxley was probably the first.

In part it was only possible because they worked on the squid “giant axion” which was sufficiently physically large that electrical activity could be stimulated and measured.

However their model was in effect only a tiny part of the story. In electrical terms they characterised the “conductor propagation” properties and it’s ability to work as a controlled damped “relaxation oscillator” to signal.

In short it was only one of the three out of

1, Communications
2, Storage
3, Processing

That is their model covers only the mechanisms that give rise to the generation of the “action potential” or “spike” and the channel characteristics that form the primary high-speed “communication” of information. But not what goes on at the “synapse” where not just slow communications but other aspects of “storage” and “processing” take place.

Whilst the biological details of the neuron are to put it politely “difficult to get your head around”, the mathematics abstracts much of this away.

The downside of this is much of the biological potential is removed and you get left with the multiple-input single-output model which is what the DSP MAD structure builds.

But the shrug of the shoulders and “so what” response happens because as noted most of the information handling in the brain is not in the neuron cell, but in the chemical activities in the synapse junction where the spike signal is converted to a chemical action as the output signal from a neuron that is then chemically coupled into the input to the interface of the subsequent neuron where it is converted into a new spike signal.

To say synapses are complex and the full chemical-biological detail of them is still unknown would be a bit of an understatement. Because whilst some fraction of their characteristics have been superficially put into mathematical formulations. The abstractions used are such that they fail to capture the majority of the biological synapse function. And what they do model they do so with at best mostly low degrees of accuracy.

As I’ve mentioned in the past there is in reality no such thing as “digital electronics” in the way most think of logic gates etc. The 1973 McMOS book from Motorola goes into quite some depth of how they are in reality “Analog amplifiers with open loop high gain” that get driven into the rails. The important thing where “most fear to tread” is the issue of “Metastability” which I’ve also mentioned in the past.

Back in the 1980’s the VLSI guru Carver Mead gave a lot of thought to this. As a result he pioneered quite a number of developments in what was called “bio-inspired microelectronics”. One result was the 1989 publication of his book

“Analog VLSI and Neural Systems”
(Addison-Weasley ISBN:978-0-201-05992-2)

https://archive.org/details/analogvlsineural00mead/mode/1up

It’s a book you might find of relevance as many consider the work in it to be the foundation stones of neuro-electronics we now call “Neuromorphic Systems”.

He is at pains in the book to separate out the neuron and the synapse and with good reasoning.

Since you brought it up the basic DNN is actually not Turing Complete but it is certainly a state machine. Most LLM DNN networks are not as such Turing complete either nor are they run as such or even designed to be.

The reason, as configured they are not capable of recursion or the ability to “write to the tape” storage which forms the multiplier weights.

This falls as such to the “transformer” used during training and where included the “adaptive” mechanism that is all to frequently referred to in the press as ML.

Why you brought it up I’m not sure, because being Turing Complete does not as such bring anything particularly new to the table. Also it is something that emerges in most systems where,

1, There is updatable storage.
2, There is a feedback mechanism.
3, There is a storage update mechanism within the state machine actions.

As was shown some years ago in the paper

“mov Is Turing-Complete”

by Stephen Dolan from Ross Anderson’s Cambridge Computer labs,

https://harrisonwl.github.io/assets/courses/malware/spring2017/papers/mov-is-turing-complete.pdf

The Intel iAx86 processor has a “Ghost Turing Engine” due to the way the memory addressing system works …

Yes it could be used as a Turing Engine but it was about the same level of usability as

https://theoutline.com/post/825/brainfuck-coding-languages

And similar “esoteric minimalist languages”. In his paper Steve laid down a challenge to write a compiler that only output mov instructions. And yes somebody took up the challenge, Chris Domas wrote a couple of compilers, and things have since become interesting,

https://esoteric.codes/blog/movfuscator-and-reductio

As it has major security implications.

Because not all such languages are effectively “purposeless”…

As I’ve mentioned before for my sins back “a long time ago in a place far, far away…” I designed a serial computer that was 1 bit wide. A part of which was a highly optimised and fast serial adder. As such it’s instruction set was very small and it had a fairly specific purpose to replace a “ladder logic” control system. So not unlike why the Intel 4004 was designed for a calculator,

https://en.m.wikipedia.org/wiki/Intel_4004

So yes, with the three requirements for filled even a database query can be made Turing Complete.

Clive Robinson • July 6, 2024 10:01 PM

@ Winter, Jonknowsnothing, ALL,

Re : Spectrums of vectors, and Filters.

“LLMs extract the “meaning” of a phrase from all the contexts it has been found in and the current context it has to be interpreted in.”

Err no the LLM does not. At a 20,000ft level it’s a two step process,

1, Build the DNN
2, Query the DNN

The LLM that most people see is the stand alone DNN embedded in a user interface.

The first step of which can be viewed as a two step process,

1, Build the vectors
2, Form the DNN from the vectors

Of which the first step can be viewed as a several step process of,

1, Pull in subset of corpus
2, Verify input
3, tokenise input,
4, Build vectors.

It’s this fourth stage where all the supposed “magic happens”. If we assume the token is a word or sub-word then it has a definition that can be turned into a vector of scalars where each scalar is a spectrum by which a necessary attribute of the token can be quantified and irrelevant or unnecessary attribute scalars set to a null value (speeds up evaluation).

The selection of the scalars of which there might be a thousand or more in each token vector is where a resulting DNN is made or broken, and contributes quite a bit to the hallucination problem or the “GO of GIGO” because it underlies the GI issue.

There is a spectrum on which the method used to select the attribute scalars/spectrums falls. One end is “Human built” the other is “statistically built”. But it is very rare to find any selection process to be one or the other because they are both “deficient” in different ways.

You might want to read a “Morning Paper” from Adrian Colyer[1] about the leading edge of research papers on what was then called NLP from a decade ago,

https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/

It will explain a couple of things, firstly why there are actual real world physical limits on the making of the DNNs in LLMs and secondly why they are not magic just Cosine/RMS systems over very many spectrums / dimensions. So just boring simple statistics not intelligence.

But a simpler and more recent over view is possibly a better place to start,

https://saschametzger.com/blog/what-are-tokens-vectors-and-embeddings-how-do-you-create-them

[1] I’ve mentioned Adrian Colyer before on this blog when he was writing his blog “the morning paper”. He used to write daily about one or more new papers in Computer Science. Like me he was based in the UK and had a one hour commute either way to fill and spent it in effect reading and note making. However due to C19 “lockdown” and other issues he had to stop which is a shame. His blog is still up and available as a resource and I would still recommend reading it especially if you are starting out on post graduate work with the aim of doing research.

Clive Robinson • July 8, 2024 3:19 AM

@ JonKnowsNothing,

Re : Fluidic computing.

You might find this of interest,

https://royalsocietypublishing.org/doi/10.1098/rstb.2018.0372

It’s a history of calculating and logic devices based around the use of fluids or fluid paths and the gradients created, moving on to even more curious devices.

One or two I’ve seen in the London Science museum like the fluid “computer” used to model the economy, others I’ve only heard about.

I was aware of the use of biological components like “slime moulds” decades ago having seen as a young lad how the mycelium of fungi spread along the underneath of rotting planks. As for the poor slime mold it became a bit of a joke amongst a group of people I used to know. And the use of ants was something that got discussed with one over an outdoor lunch, a well known author who referenced it in one of their books.

But do you remember that Scientific America article last century about DNA being used to do massively parallel computing? That was used to solve an “optimization problem” by the cryptographer Leonard M. Adleman,

https://www.scientificamerican.com/article/computing-with-dna/

But If you look carefully at Fig 3c in the Royal Soc article you will see an excuse for adults to keep “working with” Lego “for research” 😉

But sadly as with other technology Section 10 points out there is a “dark side” use for the technology.

Clive Robinson • July 9, 2024 8:34 AM

@ Winter, ALL,

Re : Magic thinking does not build functional systems.

You say,

“When you reduce a system to its components, you are able to study and understand the components. But to understand the system, you have to reassemble the components into a system.”

What you conveniently forget to mention is that the assembled system can not do more than the foundation components allow.

Which is a point you apparently appear to be not cognizant of or are trying to ignore.

The basic DNN in LLM’s does not possess a system that allows recursion as I’ve explained, so it is not “Turing Complete” nor can it be, which debunks your earlier comment to that effect.

That is why the basic DNN in some systems is augmented with a “transformer layer” that gives the feedback mechanism that can adjust the DNN weights just as a DSP “adaptive filter” does. It’s this that gives some semblance of ML capability. But it’s certainly not “intelligence” or the claims originally espoused in the AGI babblings.

Which is why this comment of yours is wrong,

“Nothing you write about the components of a LLM allows us to understand why a LLM can converse coherently and answer questions in a seemingly natural, and often even factual correct, way.”

Actually they can “appear to” depending on what the training data held, and the questions you ask.

But ask a question outside of the training corpus and that’s when you get “hallucinations” as they are incorrectly called but as more recently explained is “soft bullshit” (yes BS is now a “term of art” in a scientific “research domain”).

https://link.springer.com/article/10.1007/s10676-024-09775-5

There is that “adaptive” difference between LLM and ML which is why I and others treat them distinctly and separately.

I can explain why an ML system can as you put it “converse coherently…” But I’m not here to write massive monologues.

Further I can explain why you are being “bedazzled and beguiled” rather than just understanding what LLMs and ML systems do.

But first you need to understand and accept the fallacies behind the Turing Test, and the implication of Searle’s Chinese Room and what it demonstrates.

Because at the moment you are applying “magic thinking” to LLM and ML systems of the sort that in effect says,

“If we make it complex enough intelligence will happen as an emergent property.”

It won’t as indicated because the base foundation components have to allow it. And currently the DSP arrangements in DNN’s being used will not allow it.

Worse that kind of thinking is the same as,

“If we can not explain it then God must have created it.”

And we’ve seen what kind of nonsense that causes, which a study of history clearly indicates is a very bad way for mankind to go.

Will we get Artificial Intelligence, I would say,

“Yes, but not down the cul-de-sac we are currently walking into.”

Clive Robinson • July 11, 2024 5:17 AM

@ Winter,

Re : Incorrect thinking does not build functional systems.

“It is the inference part that is not recursive”

You might want to think that statement through again.

Because by definition

“An inference is an idea or conclusion that’s drawn from evidence and reasoning. An inference is an educated guess.”

https://www.vocabulary.com/dictionary/inference

An inference is implicitly a recursive process as it’s based on “reasoning” which is a recursive process and “evidence” which in knowledge gained by tested evaluation which is again a recursive process.

Likewise “educated” is a process of learning by “training” which is again a recursive process. As you yourself point out,

“Also, LLM training is recursive, albeit somewhat indirect by error back propagation.”

By the way that “error back propagation” is what the “Transformer Layer” added to a DNN carries out, and also in a more general sense what those less than a couple of a dollars an hour humans do on detected errors.

Clive Robinson • July 25, 2024 7:24 AM

@ Moz, ALL,

Re : End of a freedom.

With regards,

“This is a security failure. We have been defeated by those that wish the internet and even humanity ill.”

I saw it coming “in a way” in what was happening years ago. It started with certain people making personal not objective criticism against an individual as a singular agenda. But never making anything approaching a societal contribution that was constructive, relevant, or original.

Their intent back then was to damage reputations in the eyes of casual readers[0]. This inturn encouraged others to “go scalping” for what I assume was petty bragging rights in small cliques of mostly irrelevant interest to the greater majority.

For a while such behaviour was popular and mostly based on fake comments that got drummed up by others “looking for topics” to write up or gain their 15 seconds of fame. It did a lot of harm for a while untill others started to realise it was possible for those of ill intent to create smoke and noise without there actually being a fire. But in the process they did burn a lot of people.

The supposed right of “free speech” is a precious gift to be used wisely. Like all tools it is a double edged sword that can be used for good or bad. Misuse to polish a personal ego or gain fame or notoriety via a baseless agenda –which is what all to many have tried to use it for, often infamously so– damages free speech with each such use in the eyes of society.

Which is why over the years Judges have decided there are limits to “free speech” which is why even school kids used to get told “Shouting fire in a crowded auditorium” is not free speech. In part because of the considerable harm it does to innocent people.

The principle behind this cautionary behaviour is much as it is with early professions of “First do no harm”. Unfortunately the lack of dealing with such bad behaviours led upto the point it “became a thing” of Social Media one name of which was “Cancel Culture”[1].

Whilst originally providing an outlet for those with a genuine need to “make public” which Free Speech alows, it has quickly become not just tainted, but weaponised for politics and personal agendas as will no doubt become abundantly clear over the next few months.

Whilst “political idiots” / “talking heads” espouse the power of AI to “moderate” they clearly do not understand the fact that the current LLM and ML systems really can not do such a job currently, and nor are they ever likely to. Because the LLMs are not in any way “intelligent” nor are they in any way “predictive” just grossly laggingly “responsive” at inordinate cost/harm.

But still worse the cost of keeping these “Deep” “Digital Neural Networks”(DNNs) even remotely upto date via ML is inordinate resource hungry almost beyond measure. Hence the use of the less expensive “sweat shop labour” to add layer after layer of filtering, which actually makes the resource usage issues worse.

Whilst it was clear to me that the “agenda / fame” nonsense was going to rise and I gave repeated warning, it turns out others were thinking in this area both more intently and academically. One such is a fellow at the Berkman Center, Dr. Aaron Shaw[3] who back over a decade ago in 2012 had a paper published,

“Centralized and Decentralized Gatekeeping in an Open Online Collective”

https://www.researchgate.net/publication/258174798_Centralized_and_Decentralized_Gatekeeping_in_an_Open_Online_Collective

Even though based on observations of behaviours back in 2008 it is still an interesting read on the non technology effectively social aspects of the subject of moderation. From which it can be seen why even other forms of AI that are not the current LLM or AI systems or those likely to spawn from them will not function as effective moderation either.

Which should not be at all surprising, as the past several millennium of human history shows trying to “social censor” against society, no matter how draconian the enforcement, in the end always fails to human ingenuity be it by the like of in channel satire, innuendo, or more subtle means. Or the simple fact control on one channel is insufficient as like ink on a wet page information bleeds across from line to line and even from page to page.

But all that aside learning is a two way process, because multiple view points give rise to thoughts, considerations and questions that might not have otherwise arisen. Thus this blog has been very much a collective effort. And for those that have read our hosts writings can see the influence this blog has had in turn on him, and other well know academics. Some of whom, Ross J. Anderson and Nicholas Weaver were regular contributors with others like Moti Yung reading and occasionally commenting.

The thing is, as was once noted, the simple and innocent question of a child if considered can give rise to a lot of deep thinking. Such as “Why is the sky blue?” or “Why are clouds white?” or the all time favourite subject of many children “What is a rainbow?”. If we truly know the answers then we can tell or show children the answers in ways that will encourage them to understand more themselves and so encourage others. Whilst the world appears to be magic, that is superficial and a deep understanding actually does not rob the world or individual of charm, it enables them to go on and create magic for following generations.

Sadly the “new policy” of the likes of Google is not to make the world available to all any longer. They have decided that you will be led astray by their choices, unless you know enough to be able to push through their bias to what is out there.

What ever their supposed “corporate reasoning” we can make an assumption it has actually to do with that triad of “Money, Politics and Power” that act as the tools of control. It actually brings forth all the human evils we associate with Kings and Bishops and their unsavoury efforts to keep people in oppression and subjugation, not for the good of society no matter how much they might claim, but for the evils that they so revel in.

This blog has acted as a place of knowledge where the triad of tools got ignored, and the control others sought not just questioned but challenged and railed against.

[0] And now of course that this blog has been scraped into current AI LLM systems apparently that ill intent has spread out in a way “search engines” could/did not previously do.

[1] In her 2022 essay / e-Book “Cancel Culture: A Critical Analysis”,

https://link.springer.com/book/10.1007/978-3-030-97374-2

Academic author Eve Ng[2] pointed out in it, her definitions of the terms “Cancelling”, and “Cancel Culture” that both are the practice of nullifying or cancelling someone or something by a process of “speaking out” or “shaming”. That is in someone’s eyes they have done “bad” in some way, hence the speaking out about them. Be they an individual, a group of individuals, an organisation, commercial or other brand, or as we are currently seeing even entire nations. By so called “Public Shaming” in what is often not a fair forum, and seeking to dictate the surrounding commentary about the alleged bad / wrong doing.

In short what first started out as a “social good” has quickly become a weapon of politics and in many places a “social bad”.

You can get a feel for the drivers behind the essayt e-Book with,

https://m.youtube.com/watch?v=95JuPhRjDn4

So to be honest I don’t recommend it as a general read because it becomes somewhat tainted and biased by what reads like cognitive bias or a cherry picking process.

[2] Associate Professor Dr. Eve Ng was at the time of writing her e-Book / essay[1], based in the School of Media Arts and Studies at OHIO University USA. As well as the associated Women, Gender, and Sexuality Studies programs with a focus on internationalisation of minority groups in small nations via online means to find/make creative spaces.

She has the problem that there is a computer security suite “EVE-NG” that coincidentally shares her name, which makes searches via the “main engines” a bit difficult.

[3] Professor Aaron Shaw based at the Department of Communication Studies at Northwestern University and also a fellow at the Berkman Center for Internet and Society at Harvard University.

As part of his research into the dynamics of large online communities and the participation, mobilization, and organization of individuals and groups surrounding them back in 2008 he published in 2012 the paper,

“Centralized and Decentralized Gatekeeping in an Open Online Collective”

Which details some of the issues of what we would now call “Comment Moderation”.