Technology

59161 readers

2294 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

[email protected]

594

AI worse than humans in every way at summarising information, government trial finds (www.crikey.com.au)

submitted 2 months ago by [email protected] to c/[email protected]

124 comments fedilink hide all child comments

Artificial intelligence is worse than humans in every way at summarising documents and might actually create additional work for people, a government trial of the technology has found.

Amazon conducted the test earlier this year for Australia’s corporate regulator the Securities and Investments Commission (ASIC) using submissions made to an inquiry. The outcome of the trial was revealed in an answer to a questions on notice at the Senate select committee on adopting artificial intelligence.

The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.

Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.

These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.

(page 2) 50 comments

sorted by: hot top controversial new old

[–] [email protected] 21 points 2 months ago (2 children)

I would expect "faster" to be a way

[–] [email protected] 7 points 2 months ago

"I can easily do it on my phone" is also good.

[–] [email protected] 12 points 2 months ago* (last edited 2 months ago)

or cheaper

[–] [email protected] -2 points 2 months ago

Artificial intelligence is worse than humans in every way

As if capitalists have ever cared about that...

[–] [email protected] 1 points 2 months ago (1 children)

No shit.

[–] [email protected] 0 points 2 months ago

Intelligence vs non intelligence: intelligence is superior... Who would have thunk it lol

[–] [email protected] 27 points 2 months ago (3 children)

Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts.

This is the key line here. These are likely university educated staff with significant experience in writing and summarising information and they were specifically tasked with this. However, within the social media landscape (Lemmy, reddit, etc) AI is already better at summarising information than humans because most human social media users are fucking retarded and spend their time either a) not reading properly/at all or b) cherrypicking information to fit whatever flavour of impassioned narrative they are trying to sell to everyone else.

Just some very recent examples I've seen of Lemmy users proving they are completely incapable of parsing relevant information are that article about an alternative, universal and non-proprietary database called GetGee which everyone seemed to think was an article about whether TikTok should be banned (because the word TikTok was in the title and that tricked their monkey brains) or the update to the 404 Media story on "active listening" in which people responded as if this technology exists and is in use when 404 Media still haven't been able to confirm either of these things. The second one was particularly egregious because it got picked up by all kinds of tech-related YouTube channels and news sites and regurgitated by their viewers and readers without a single one of these people ever bothering to read the source material properly.

[–] [email protected] 8 points 2 months ago

Lemmy users proving they are completely incapable of parsing relevant information

To be fair, you need to actually read the article to be able to summarize it.

load more comments (2 replies)

[–] [email protected] 2 points 2 months ago

Nice to have though, would likely skip or half-ass a lot of stuff if I didn't have a tool like AI to do the boring parts. When I can get started on a task really quickly, I don't care what the quality is, I'll iterate until it meets my standards.

load more comments