Maybe it’s because I’ve been reading a Bruce Schneier book lately (he’s a security expert), but I think that the Singularity Institute for Artificial Intelligence is facing a very real threat.
It is easy to imagine at some point in the future, when the Institute is well into the implementation phase, that many countries will consider AI a national security issue and that they will use their intelligence agencies to spy on work done in the field.
It can be extremely hard to defend yourself against such well funded attackers; they can try to steal or hack hardware or software, bug your offices and homes, infiltrate your team, bribe employees, use social engineering, make things look like an accident or petty crime, etc.
One tactic might be to wait for the Singularity Institute to be almost done (in the pre-launch testing/auditing phase, for example), steal the code and throw a lot of resources at it to be the first who launches a recursively improving artificial general intelligence. This could lead to disaster if whoever does this does not have benevolent intentions or is not as careful as the Institute would be.
My recommendation to the Singularity Institute is to make sure to have top security experts on the team and to prepare well in advance for the future when security might become critical.
July 2, 2007 at 3:46 pm |
Most of the higher-ups (and the population in general) don’t believe general AI will be possible for many, many decades, if not centuries. So the threat is not that great unless we accidentally convince them that AI is possible in the closer term.
July 2, 2007 at 7:32 pm |
[...] and you should consider checking out the latest posts. On his own blog, Michael Graham Richard considers the security risks that a late-stage AGI project might face. My response: if AGI researchers [...]
October 5, 2007 at 12:21 am |
“So the threat is not that great unless we accidentally convince them that AI is possible in the closer term” – Michael Anissimov
Which puts us paradoxical situation.
Also, doesn’t belief that a threat is if low probability, rise the probability of threat.
October 30, 2007 at 2:01 pm |
[...] Yudkowsky, a research fellow over at the Singularity Institute for Artificial Intelligence (see my previous post warning them about security issues), asks a very interesting question over at Overcoming Bias: [...]