One can't have an informed opinion on this without witnessing how grading a college exam takes place, first hand.
Grading a stack of blue books is a "just kill me now" brutal experience. A majority of cognitive effort is just finding the page for the next problem to grade, with finding the answer another delay; any programming language with this kind of access latency would stay a "one bus" language. So of course professors rely on disinterested grad students to help grade. They'll make one pass through the stack, getting the hang of the problem and refining the point system about twenty blue books in, but never going back.
With stapled exams one problem per page one can instead sort into piles for scores 0-6 (if you think your workplace is petty, try to imagine a sincere conversation about whether an answer is worth 14 or 15 points out of 20), and it's easy to review piles.
When I had 200 linear algebra exams to grade at once, I'd scan everything, and use my own software to mark and bin one question at a time, making review passes a pleasure. I could grade a 10 question final in one intense sitting, with far more confidence in the results than team grading ever gave me.
spicyusername · 3h ago
If learning is the goal, and we just learned that we need to do more work to provide an experience conducive to learning, then the cost for learning just increased.
First, its interesting to see a situation where a new technology INCREASED the cost of something.
Grading a stack of blue books is a "just kill me now" brutal experience.
When I had 200 linear algebra exams to grade at once
Second, this is alluding to one of the dimensions along which that cost is going to be born, staffing. If its too many papers to grade, then more staff are needed to grade them.
We're probably not going to get that, though, and instead the cost increases, the funding stays the same, so the quality will decline to compensate.
So it goes...
ipcress_file · 8h ago
I think problem number one is that you had 200 exams to grade.
kkylin · 5h ago
This is not an uncomon situation in e.g., math departments across the US. Luckily I've not had to grade more then 60-70 in a single term, or ~200 but a select number of problems in a large course.
Also, second stapled exams with prescribed spaces for answers. So much time wasted just looking for where the answer is.
downrightmike · 5h ago
The children yearn to learn. And the admin cuts costs.
Balgair · 2h ago
You make a great point here. The quality of the grader may be higher with an AI than a stressed out grad student.
It's long been known that a longer essay answer is more likely to get a higher grade. And yeah, having been a student and a stressed out grad student, after about the 20th exam, only length is the real signal of grade.
Other comments point out that with the prices of tuition these days, students should be expecting a lot higher quality of feedback (grading) than what they are getting at any random R1.
It really does seem that the University system (as opposed to the college-esque system) is broken and that the additional AI fears are just another log on the already collapsed bridge. We're getting over the wrong thing.
mlpoknbji · 7h ago
This problem is solved by software like gradescope. Makes grading extremely fast and much more consistent (because of easy rubric adjustments on the fly). This is more concerning a STEM exam, admittedly I don't know how well this works on humanities essays.
kkylin · 5h ago
Gradescope partially solves the problem. There are certain kinds of questions that are more amenable to rigid rubrics (e.g., techniques of integration in a beginning calculus course) where the space of correct solutions is rather limited. Once the material gets more advanced, the space of correct (and partially correct) solutions gets really large, fast. One could design exam questions to restrict the solution space, but sometimes at the cost of testing real understanding. Also, at that point giving partial credit with Gradescope-style rubrics is not always straightforward (not impossible, but not trivial).
armchairhacker · 8h ago
It doesn’t have to be a book, it can be a cheap laptop without internet.
teeray · 4h ago
The idea is the same: society needs AI-free enclaves to measure aptitude. How those are realized can vary, of course.
smelendez · 4h ago
Probably someone will build and sell a dirt cheap laptop with nothing on it except a basic word processor for this purpose.
It’s still a hard problem though. If the students have the laptops outside of the testing site, they can load cheating materials on them, or use them to smuggle questions and answers out if the test is used in multiple class sections. You realistically will not lock down a laptop students can take home sufficiently that some people won’t tamper with it.
Otherwise you have to have enough laptops to get each student a wiped and working machine for every test, even with lots of tests going on. And students need to be able to plug them in unless the batteries are rigorously tested and charged, but not every classroom has enough outlets. And you need to shuttle the laptops around campus.
Then you need a way to get the student work off the laptops. You probably want the student to bring the laptop up when done and plug in a USB printer. Anything else, like removable media, and you have to worry about data loss or corruption, including deliberate manipulation by students not doing well on the exam, and students claiming what got graded wasn’t what they meant to hand in. And you still have to worry about students finding a way to deliberately brick their laptops and the inevitable paper jams and other hardware failures, especially an issue when students need to leave on time to get to another class.
So you need systems that are cheap but reliable, tamper-resistant but easy to diagnose and maintain, robust against accidental data loss but easy to reliably erase, able to export data and install security updates without letting students surreptitiously input data, and heavily locked down while easy to use for students with a wide variety of backgrounds, training, and physical abilities.
armchairhacker · 2h ago
Store in each classroom a custom-made rack for the laptops (with at least one laptop per seat plus backups). Distribute the laptops at the start of exams and collect them at the end.
The rack charges the laptops, streamlines distributing/collecting tests, prevents tampering, and reports defects. A professor uploads the test to their LMS and specifies the exam time-frame and location. Before the exam, the LMS transfers the test to the correct rack, which saves it to the laptops. After the exam, the rack loads the student responses and transfers them to the LMS for the correct class. Each laptop has a light under it whose color indicates whether it's in standby, waiting to be distributed (shortly before exam start), has a submitted test, or is malfunctioning (the rack periodically pings laptops in storage to verify they still work). Multiple professors can queue exams to the same location, in different time-frames, and the LMS and rack know when to prepare each test on the laptops.
The hard part is to implement this well. The rack's hardware and software must be reliable; nonetheless it will fail (hardware breaks), and exams get moved and rescheduled, so there must be a way to transfer tests to another classroom and time-slot. The laptops can have slightly less reliability, since there are backups; but failures can't be clustered, and if a laptop is transferred from another rack during an exam, it should download the destination rack's test and become active (so if one rack's backup laptops run out, laptops can be borrowed from other classrooms). The laptops must periodically backup in-progress tests using wifi or bluetooth, and if a laptop breaks during the exam, the student can resume their progress on another. Tests must be downloaded well before exam time, in case there are problems (laptops should be able to store and queue multiple tests). Laptops must handle exams that start late (up to the official exam end time) and end late (including overlapping the next exam's start time, in which case the next exam is loaded when the laptop is put back into the rack). The rack must absolutely not indicate that a laptop is ready with an exam if it's not; and (since tests are downloaded before exam time) nothing unique should happen at exam start time (to reduce last-minute surprises, and let a professor who isn't convinced take out a laptop and check that it's functional 15 minutes before). Last but certainly not least, the UI to submit tests and grade responses should be intuitive, simple (not overwhelming) yet powerful (can handle edge-cases like rescheduled tests, randomized tests, accommodations; or be extensible enough to support these features).
Despite all the above requirements I actually think such a product is feasible. Unfortunately, with firsthand experience of typical EdTech and academic bureaucracy (and I'm not a professor so I don't know the worst of it), I'm skeptical it would be adequate quality unless the company designing it is uniquely motivated and capable.
pimlottc · 1h ago
Or get a stack of dirt cheap blue books and pens.
kjkjadksj · 6h ago
Speaking as someone who has graded in grad school, the general advice is to look at a few questions before you start dishing out points so you can refine your internal rubric. Going back is also done because that is easier than having a student show up at office hours and ask why sally only got marked down 3 points for the same mistake on question 5.
hollandheese · 6h ago
Huh? Blue books are typically used for essays and are graded whole.
I've never seen anyone attempt blue books for anything else.
SoftTalker · 5h ago
Late 1980s blue books for almost everything that wasn’t multiple choice. The other style was multiple pages, with large “show your work here” areas below each question.
rubidium · 5h ago
I used them in graduate physics courses 2008-2010
ramenbytes · 5h ago
And I used them in a graduate physics course last fall.
Mountain_Skies · 4h ago
One of my history professors would watch horror movies while grading history essay exams. Have to give him some credit for not taking the easy way out with scantron multiple choice test, requiring long form essays to demonstrate understanding of not just historical events but also their impact.
mlpoknbji · 10h ago
The commenters lamenting this trend presumably have not given a takehome assignment to college students in recent years. The problem is huge and in class tests are basically the only way to test if students are learning. Unfortunately this doesn't solve the problem of AI assisted cheating on in-class exams, which is shockingly prevalent these days at least in STEM settings.
refulgentis · 6h ago
I'm curious, how is AI assistance on an in class exam even possible? I can't picture how AI changed anything from, say, post-iPhone. i.e. I expect there to be holes in security re: bathroom breaks, but even in 2006 they confiscated cell phones during exams.
I guess what I'm asking is, how did AI shift the status quo for in class exams over, say, Safari?
i_am_proteus · 6h ago
People bring in second phones: one to have confiscated, one to use on the exam.
A common mode I have seen is phone in lap, front-facing camera ingests an exam page hung over the edge of the desk. Student then flips the page and looks down for the answer.
djoldman · 5h ago
It's relatively straightforward to immediately suspend test takers for a semester and expel them on a second infraction.
Administrations won't allow it because they just don't care enough. It's a pain dealing with complaining parents and students.
In any case, cheating has existed since forever. There is nothing much new about cheating in in-class exams now with AI than before without.
refulgentis · 4h ago
I am amenable to this argument (am GP), however I do think this is unique.
AI is transformative here, in toto, in the total effect on cheating, because its the first time you can feasibly "transfer" the question in with a couple muscle-memory taps. I'm no expert, but I assume there's a substantive difference between IDing someone doing 200 thumb taps for ~40 word question versus 2.
(part I was missing personally was that they can easily have a second phone. same principle as my little bathroom-break-cheatsheet in 2005 - can't find what's undeclared, they're going to be averse to patting kids down)
pragma_x · 5h ago
Seems easy to thwart with two exam proctors, one at each end of the exam room. Hypothetically, the whole affair is going for maximum manual effort (grading), so why not double down on test-taking labor too?
bigmattystyles · 5h ago
Kaplan (?) seems to have this figured out. When I went in for GREs 15ish years ago, it felt like going into a minimum security prison.
I could see those being popular as interview centers. Big company pays some company certified to provide AI free certified interview rooms.
everybodyknows · 24m ago
Why would anyone downvote this thoughtful question?
djoldman · 4h ago
It's hard to have any substantial sympathy for the troubles of teaching private US college and university students.
Given the absolutely wild increases in tuition, administrations should have massive resources to bring to bear to solving this and other problems.
The reasons they don't feel more like excuses than legitimate explanations.
secabeen · 3h ago
The wild increases in tuition are for the nameplate price, not the actual price paid by median students net of grants. See pg. 12 here:
Published tuition has gone up, but grant aid has matched it, making the net cost at 4-year public institutions to be flat, or even down slightly over the 10 year period. The same applies at private 4-year institutions, large increase in nameplate price, matched by large increase in grant aid, actual net tuition flat.
Expenditure data also show that they are not spending significantly more. See the chart at the end of this page, which gives expenditures in real, per-student dollars. They are up, a little less than 20% over 10 years, but half of that is increases in hospital costs of their associated health care systems, which have nothing to do with tuition. The rest is a mix of various activities of universities, much of which are not tuition-funded.
Interesting. I'll have to dig into this more. Thanks for the links.
acbart · 3h ago
No no, that money goes to the administration. They are not involved in the teaching. That is left to the faculty, who are paid inverse to the amount of teaching they handle. Teaching is an afterthought at universities, the primary activities instead are research, building football stadiums, and paying for the revolving door of administrators.
bradleyjg · 3h ago
These institutions are not about teaching students. That’s a fundraising exercise for their actual purpose. And even the actual purpose is under heavy stress from cancerous administrative growth.
wrp · 9h ago
Survey question: To what extent did blue book exams go away?
When I started out (and the original Van Halen was still together), blue book exams were the norm in humanities classes. I've had narrow experience with American undergrad classes the past 25 years, so I don't have a feeling for how things have evolved.
ipcress_file · 8h ago
I've never stopped using them, with the exception of the pandemic year, when we were forced to run online exams.
Why replace a system that generally works well with one that introduces additional potential problems?
MikeTheGreat · 8h ago
Same here.
Online instruction / learning can work for some people, and that's good.
I don't understand how anyone ever thought that an online exam could be made secure. There's just no way to ensure that the person who registered for the course is the one taking the exam when you don't control anything about the hardware or location, when students have a wide variety of hardware that you must support, and any attempt at remove video monitoring of the exam immediately runs into scalability and privacy issues. Like, even if you're watching a video of the person taking the online exam, how do they prove that they didn't just hook up an extra keyboard, mouse and (mirrored) monitor for person #2 to take the exam for them while they do their best to type and/or mouse in a convincing way?
It also doesn't help that you periodically get students who will try to wheedle, whinge, and weasel their way into an online exam, but then bomb the in-person exam (it's so strange and totally unrelated that they really, really wanted to take an online exam instead of in-person!).
Ok, I'll stop ranting now :)
nlawalker · 2h ago
Chiming in here just because I happen to have taken a handful of online-proctored certification exams through Pearson OnVUE recently.
The gist of it is that I think someone willing to put a lot of work in could probably cheat using the strategies you suggest, but it would be a pain. During your checkin you have to send a selfie, photos of the front and back of your photo ID, and four photos of the space you've prepared. You can't have any writing tools, written materials, or anything else that looks like a computer or screen in the area, and the machine you're on has to be single-screen. If the pre-test greeter or the proctor aren't satisfied with what they see they can ask you (via text and voice chat) to show them the room in real time via your webcam and may ask you to make changes or move things around to provide evidence that something in the room is not hiding mechanisms used to cheat. From that point on, your webcam and mic are on and live streaming to the proctor for the duration of the test; they don't say anything about assistive technologies on their end but I assume they are using eye tracking to look for instances of eyes wandering offscreen for a protracted period of time. The test environment software effectively "takes over your PC" during the test and I would imagine is pretty effective at detecting alternate/multiple display outputs etc.
There are probably scalability issues, but privacy is not an issue from the perspective of the proctor - you are effectively surrendering it by agreeing to take the test online, you could have gone to a test center instead.
ipcress_file · 8h ago
I get it. My major concern was that students were cheating online (nearly 50% if I was detecting them all) who I didn't think would have cheated in the classroom. I didn't like the idea that we were creating a situation that enticed students to cheat.
That being said, the whole experience had an impact on my generally optimistic view of human nature.
msgodel · 5h ago
The way you do online exams is by contracting other universities proctored test centers and making students attend the nearest one on exam day.
There's a whole system for this, it already works very well if people actually wanted to make online exams work. Of course it's not "social distancing" so it didn't help with covid.
BrenBarn · 8h ago
I'm not sure, but I used them as recently as 2019. (Well, not a blue book per se, but a printed exam where students handwrote their answers on the provided sheets.) I'm pretty sure some faculty never stopped using them except when forced to by online classes. (Incidentally, increased pressure to offer more online classes is a worrisome combination with increased AI-aided cheating.)
jccalhoun · 8h ago
I was an English major and Math minor in the 90s and I used them for one class - a history general studies requirement. All the essay tests in my other classes were on letter sized paper with blank space for us to write in.
wrp · 8h ago
You raise a good point. I've assumed that "blue book exam" is used as a generic term for in-class tests that consist of just essays, not necessarily using the physical Blue Books that you buy at the campus bookstore.
sarchertech · 10h ago
This is the obvious solution to chatgpt essays. You could also just add more lab sections to CS classes, and force students to do assignments with no access to AI.
When I took physics we had weekly 3.5 hour lab sections. That should be enough for most CS assignments.
SoftTalker · 10h ago
The lab sections were enough time to run the experiments. Writing up the lab report took additional time outside of that, at least that's how my lab sections went in school.
girvo · 7h ago
Yep, absolutely the case for my labs.
nxobject · 10h ago
As someone whose college papers always went through multiple drafts, I genuinely hope that the "joy" the author feels doesn't get in the way of teaching writing skills.
And, as someone who got paid minimum wage to proctor tests in college, I couldn't keep a straight face at this:
> The most cutting-edge educational technology of the future might very well be a stripped-down computer lab, located in a welcoming campus library, where students can complete assignments, by hand or on machines free of access to AI and the internet, in the presence of caring human proctors.
I think the author's leaning heavily on vibes to do the convincing here.
sarchertech · 9h ago
We had a math lab where you could go do your math homework and the majority of the student assistants were great. Same with the undergrad lab assistants in physics labs.
op00to · 5h ago
Writing labs exist. What’s the existence of tutoring and assistance have to do with anything?
sarchertech · 2h ago
The OP was saying that paid student proctors couldn’t possibly be caring. I pointed out that most of the student assistants in similar positions were in fact very caring.
nxobject · 2h ago
I've done both – in my undergrad I staffed a drop-in help night for a required freshman comp class, and later on in grad school I made extra cash proctoring a "alternate testing center" for students needing retakes and ADA accommodations. I can see the comparison to drop-in homework help centers. But as a test proctor my job was to get out of the way of students as much as possible while keeping things quiet – I'm not sure it's worth idealizing as "caring".
pessimizer · 8h ago
> I think the author's leaning heavily on vibes to do the convincing here.
I have no idea what you're trying to express in your comment, so who's using vibes?*
Were you triggered by the word "caring?" A waiter usually cares that the people they're serving have an enjoyable meal. It doesn't mean that they love them, it means that they think the work of feeding people is purposeful and honest (and theirs.)
-----
[*] It's certainly not in the words; I don't know what made you angry about "joy," I don't know why you think the author does not teach writing skills in "communications," I don't know why the fact that you went through multiple drafts in writing school papers is relevant or different than anyone else's experience. Maybe that's over now. Maybe I actually don't care if you use AI for your second and further drafts, if I know you can write a first draft.
nxobject · 7h ago
Why would you assume I was angry? (Where is your snark coming from?)
Drafting and redrafting a cumulative course paper, as well as iteratively honing a thesis, is a writing skill.
I would argue it as important than demonstrating recall and interconnection of the material. It is being lost if long-term work on papers is being replaced with 3-hour blue-book essays.
That is why I thought it was relevant. That's it.
CompoundEyes · 9h ago
There are schemes happening in interviews too. I've been doing many technical interviews for a senior role with off shore candidates lately. They come to me after passing a challenging online test that has controls to check for browser focus lost, opening new tabs etc. I think it may require a camera to be on too. Even with that more than half of the candidates can't code.
Our interview usually starts with them breathlessly reading from a script out of the corner of their eye. I'm ok with notes to make sure you hit some high points about yourself even in person. Nervousness shouldn't disqualify a talented person. But with the coding part I've gotten exasperated and started asking these senior candidates to share their screen and do a fizz buzz exercise live in a text editor in the first few minutes. If they struggle I politely end the interview on the 15.
One candidate cheated and it was interesting to watch. In the time I sent the message in Zoom and them sharing their screen, just a few seconds, they had either queried or LLM-ed it on their phone or another computer, had someone off screen or in the same room listening and sharing the answer on another monitor or something else. Whatever it was they turned their head slightly to the side, squinted a bit and typed the answer in Java. A few syncopated characters at a time. When asked what modulo was they didn't know and couldn't make any changes to it. It was wacky. In retrospect I think it was them reading the question out loud to an LLM.
I'm waiting for the candidate who has someone behind them with the same shirt on pretending to be their arms.
lispisok · 5h ago
Have you considered using onshore candidates instead?
Mountain_Skies · 4h ago
Management would prefer that cheap labor burn the company to the ground than pay a fair wage. It's something more perverse than simply saving money.
OptionOfT · 7h ago
> They come to me after passing a challenging online test that has controls to check for browser focus lost, opening new tabs etc.
These are the absolute worst.
You're taking people out of their comfort zone (highly customized IDE like JetBrains / VSCode / Vim) which cause them to lose shortcuts and decently working intellisense. Yes, my TypeScript in my projects is configured in such a way that I get way more information from the compiler than the standard config. After all, you're testing my ability as a software engineer, not a code monkey, right?
In this very uncomfortable place there is no way of asking questions. Yes, sometimes stuff is ambiguous. I rather have someone who asks questions vs someone who guesses and gets it right.
The testing setup is horrible too. No feedback as to what part of the tests fail, just... fail.
No debugger. No way of adding log messages. When was the last time you've been in that situation at your workplace?
All under the pressure of time, and additional stress from the person that they really NEED a new job.
Oh, and when you use compiled languages, they're way slower than say TypeScript due to the compilation phase.
And then even when your score (comprised of x passed tests and y failed tests) is of passing grade there is a manager out there looking at how many times someone tabbed outside of the window/tab?
Which reminded me that one time I used a function recently stabilized, but the Rust version used was about 8 versions behind. With that slow compilation cycle.
/sigh.
twic · 6h ago
The flip side is that the expectation is not very high. The interviewer knows you're not working in a proper professional coding environment, so they don't expect proper professional code. They do expect you to be able to produce basic working code without those tools, and to think about the problem, which does not require tools.
spike021 · 5h ago
>The interviewer knows you're not working in a proper professional coding environment, so they don't expect proper professional code
Everyone says this over the years, even before AI, and I've never felt it made the slightest difference in how they rate me.
pragma_x · 5h ago
FWIW, that's about where the bar is set with live coding exercises during a job interview. I've done this, both at a whiteboard, and online in a provided web sandbox.
I'm all for written exams, but not on the traditional physical Blue-Books. Just allow students to use traditional paper and then staple the exam to their answers as they're turning it in.
On the few occasions (3) I took one, I ended up "punching through" the paper every single time. I tend to write with high pressure and the paper quality is atrocious. Twice, the tares were so bad the book was partially de-bound.
On both occasions, when presented with a "torn/destroyed book" I had to show the proctor the "issues" and then, very carefully, hand-copy over everything into a new book in their presence--absolute PITA.
siliconc0w · 7h ago
The other side is with AI tutoring, you don't even really need exams. You have a constant real time map of where the student is strong and where they're weak.
xphos · 7h ago
But like you don't have that because AI will lie to you but kindly. People who are learning don't have the tools to point out those types of mistakes yet either so they will learn things the wrong way. Not to say teachers dont teach the wrong thing most of math is learning the wrong way only to later distill the right lesson but I think the automation of it is much worse
umbra07 · 4h ago
You can literally just system-prompt-fix that kind of problem though.
apparent · 6h ago
When I was in school, some older students complained that tests were hybrid: you could handwrite or you could type. They complained that they had grown up before typing was as common and that it wasn't fair to let students type, since they could write much more. They argued for a word limit, which was denied.
Now we'll see the reverse, with students arguing that they can't handwrite effectively and need more time or whatever in order for exams to be fair. Hopefully handwritten exams will become the norm from grade school onward, so that this complaint will be rendered moot.
i_dont_know_ · 10h ago
I feel like these kind of things push us as a society to decide what exactly the purpose of school should be.
Currently, it's been a place for acquiring skills but also a sorting mechanism for us to know who the "best" are... I think we've put too much focus on the sorting mechanism aspect, enticing many to cheat without thinking about the fact that in doing so they shortchange themselves of actual skills.
I feel like some of the language here ("securing assessments in response to AI") really feels like they're worried more about sorting than the fact that the kids won't be developing critical thinking skills if they skip that step.
Maybe we can have
viccis · 8h ago
The current system "sorts" the students who "developed critical thinking skills" out from the ones who didn't put in effort. If there's no expectation that they'll be sorted thus, then the vast majority won't (and right now don't) bother with developing or exercising those skills. Usually they'll just put all their effort into the one or two classes they have that actually make them demonstrate mastery of the material.
belinder · 10h ago
Sure school is for acquiring skills, but it's also day care. A place to keep children during the day so their parents can work, especially in a society where more and more the expectation is that both parents work.
bdowling · 9h ago
The article is about college-level education, which is primarily about ranking students in order of who should get the best entry-level jobs. If technology is disrupting the effectiveness of that ordering function, then something needs to change.
Merrill · 7h ago
There is evidence that the ranking of students in order of who should get the best entry-level jobs is done mainly by the college admissions process which bins students into more or less selective colleges.
xqcgrek2 · 10h ago
How do blue book exams translate to real world problems the students will encounter?
Why not open book + AI use exams, because that's what students will have in their careers?
acbart · 10h ago
There are many subskills that you must be proficient in without tools, before you can learn more interesting skills. You need to know how to do multiplication by hand before you rely on a calculator. If you can't do multiplication with a calculator, you're not going to be able to make sense of the concepts in Algebra.
noosphr · 9h ago
Algebra has nothing to do with long hand multiplication, people who say otherwise can't do either.
We know, because we taught computers how to do both. The first long multiplication algorithm was written for the Colossus about 10 minutes after they got it working.
The first computer algebra system that could manage variable substitution had to wait for Lisp to be invented 10 years later.
twic · 6h ago
Baby rather than Colossus. Colossus wasn't programmable.
noosphr · 5h ago
>Jack Good, a veteran of Colossus practice at Bletchley Park, later claimed that, if appropriately configured,
Colossus could almost have carried out a multiplication but that this would not have been possible in practice
because of constraints on what could be accomplished in a processing cycle. We have no reason to doubt this,
though it would presumably have required special settings of the code wheels and message tape and been, even if
possible, a rather inefficient alternative to a desktop calculator. This fact has been offered as proof of the flexibility
of Colossus, which in a sense it does attest to: a device designed without any attention to numerical computations
could almost have multiplied thanks to the flexibility with which logical conditions could be combined. Yet it also
proves the very real differences between Colossus and devices designed for scientific computation. Multiplications
were vital to computations, and a device that could not multiply would not, by the standard of the 1940s, be
termed a “computer “or “calculator.”
The limitation seems to have been physical rather than logical.
mlloyd · 10h ago
There are also many subskills not worth learning to some people. Sometimes traversal is what's needed and not understanding. (Though I'm never going to knock gaining more understanding)
Tools allow traversal of poorly understood, but recognized, subskills in a way that will make one effective in their job. An understanding of the entire stack of knowledge for every skill needed is an academic requirement born out of a lack of real world employment experience. For example, I don't need to know how LLMs work to use them effectively in my job or hobby.
We should stop spending so much time teaching kids crap that will ONLY satisfy tests and teachers but has a much reduced usefulness once they leave school.
Are multiplication and long division by hand really necessary skills?
I never need to "fall back" to the principles of multiplication. Multiplying by the 1s column, then the 10s, then the 100s feels more like a mental math trick (like the digits of multiples of 9 adding to 9) than a real foundational concept.
benbreen · 10h ago
Exactly, this is the reason why I struggle with this sort of solution to the problems we are all facing in education currently. On the surface it seems to make sense, but the blue book exam is entirely artificial and has basically no relationship to real world skills (subconsciously, even the quality of student handwriting handwriting could influence how graders assess blue books). Even leaving AI aside, anyone writing anything nowadays is using Google and Wikipedia and word processors, so why constrain those?
Oxford and Cambridge have a "tutorial" system that is a lot closer to what I would choose in an ideal world. You write an essay at home, over the course of a week, but then you have to read it to your professor, one on one, and they interrupt you as you go, asking clarifying questions, giving suggestions, etc. (This at least is how it worked for history tutorials when I was a visiting student at an Oxford college back in 2004-5 - not sure if it's still like that). It was by far the best education I ever had because you could get realtime expert feedback on your writing in an iterative process. And it is basically AI proof, because the moment they start getting quizzed on their thinking behind a sentence or claim in an essay, anyone who used ChatGPT to write it for them will be outed.
SoftTalker · 10h ago
It really boils down to: are universities trying to teach abstract knowledge, or are they trade schools?
If they are trade schools, yes teach React and Node using LLMs (or whatever the enabling tools of the day are) and get on with it.
jbreckmckye · 7h ago
Not sure how it is these days, but at Cambridge, supervisions (what Oxford calls tutorials) did not contribute to our examination / tripos scores. They were just a learning aid.
nxobject · 10h ago
> Even leaving AI aside, anyone writing anything nowadays is using Google and Wikipedia and word processors, so why constrain those?
And the library, and inter-library loan (in my case), and talking to a professor with a draft...
lokar · 8h ago
As a CS student I rather enjoyed the blue book essay exams in my classics courses.
And it did teach and evaluate skills I’ve used me entire career.
girvo · 7h ago
For the same reason I wasn’t allowed my extremely powerful SAT solver calculator in my maths exams (for high school and first year uni where it would’ve helped):
Because I was demonstrating that I understood the material intrinsically, not just knew how to use tools to answer it.
viccis · 8h ago
Would you let a second grader take their spelling test using a word processor with a spell checker?
parpfish · 10h ago
most exams aren't about solving a problem, it's about demonstrating that you've learned something.
nxobject · 10h ago
Sadly, I think that's what's going to be lost in the switch back to exams: the ability to assess someone's ability to iteratively solve a problem or craft a thesis.
slipperydippery · 9h ago
They’re proof you learned the topic well enough to coherently write a very-few pages about it. That’s what they’re for.
Making them open book + AI would just mean you need “larger” questions to be as effective a test, so you’re adding work for the graders for basically no reason.
nine_k · 10h ago
The problem is not that a student can ask AI for answers. The problem is that the student has to have an idea what questions to ask.
For that, the student must have internalized certain concepts, ideas, connections. This is what has to be tested in a connectivity-free environment.
furyofantares · 10h ago
In fact, why teach the material at all?
altairprime · 10h ago
Can’t use AI on a date, or at a dinner party, or during a board meeting.
Faking intelligence with AI only works in an online-exclusive modality, and there’s a lot of real world circumstances where being able to speak, reason, and interpret on the fly without resorting to a handheld teleprompter is necessary if you want to be viewed positively. I think a lot of people are going to be enraged when they discover that dependency on AI is unattractive once AI is universally accessible. “But I benefited from that advantage! How dare they hold that against me!”
noosphr · 9h ago
>Can’t use AI on a date, or at a dinner party, or during a board meeting.
I get the same "you won't always have a calculator with you" vibes from 90s teachers chiding you to show your work when I hear people say stuff like this.
altairprime · 9h ago
I wouldn’t equate trigonometry, which underpins the classic parabolic example you’re referring to, with critical reasoning in human conversation. One is situationally useful at best; the other is mandatory to prevent exploitation by malicious people. Mental quadratics may be appealing, but the ability to reason is the bare minimum. Besides: if you’re using a calculator or AI in a board meeting, you’re likely unprepared for the board meeting.
noosphr · 8h ago
I'm not sure how you did proofs in trigonometry with a calculator.
altairprime · 8h ago
I ran my school district out of math to teach me in the 80s and ended up focusing on sysadmin in the 90s rather than repeating trigonometry or pursuing formal math at the local college. Sorry I can’t be of more use to your argument! Perhaps someone else will have applicable experiences.
bigstrat2003 · 9h ago
You say that like those teachers were incorrect. They were correct, and still are correct. You don't always have a computer to hand, and you do in fact need to be able to do basic math.
toast0 · 3h ago
It's pretty rare that I don't have quick access to a calculator. Which really reduces the basic math I need to do in my head.
It's more likely I will not have paper and writing implements than not having a calculator.
Besides, most people have room for fast arithmetic or integrals; fast arithmetic would be more useful, but I'm not putting the time in to get it back.
Ekaros · 8h ago
Also LLMs have fairly well proven that even if you have calculator you probably should have ability to do some sanity check on the answer. In case you hit wrong button for example. With LLMs they can be confidently wrong and unless you are able to tell you are out of luck...
Plus all about capability to actually retain whatever you ask from the model...
altairprime · 8h ago
Ironically, America finally automated one of our oldest workplace specialties: grifting! People overconfidently declaring made-up nonsense in board meetings is a classic executive behavior. I suppose it’ll be interesting to see what happens now that everyone has one in their pocket. Will their patter improve from exposure alone, or will they more easily detected because their skills weaken from disuse?
noosphr · 8h ago
If I don't have access to my phone the power grid has been down for at least two days and by that point I've got more pressing issue than showing my work when doing basic math.
cyberax · 9h ago
> Can’t use AI on a date, or at a dinner party, or during a board meeting.
Cheaters are universal no matter what social boundaries are defined, and neither high-bandwidth wireless signals nor onboard acoustic processing can be reliably performed within the human rectum to any reasonable degree of fidelity. If one would externalize basic critical reasoning skills, I encourage finding another location to store one’s supplemental cranium :)
philwelch · 9h ago
If there weren’t differences between the educational environment and the workplace, that would be the same as if you just dumped people straight into the workplace with no education. For education to be useful at all it has to be distinct from the “real world” and optimized for learning. One way of optimizing for learning is to make the student solve the problem the hard way, so the student understands the entire process and not just the parts that are harder to optimize.
add-sub-mul-div · 10h ago
Because a tradeoff of relying on calculators means the average person can't do simple math on their own. It would be bad for society if a crutch for general thinking dulls those skills the same way calculators dulled home economics skills. In theory people could pull out a calculator at the grocery store but I've never seen it happen.
repiret · 33m ago
I don't know what you mean by "home economics", but to me, that encompasses things like balancing a checkbook and making a budget and taxes and understanding how savings and debt and compound interest works how to choose when to save and when to go into debt. The sort of money matters that apply to any schmuck trying to live in the world. The reason so many people lack those skills is that for the most part we don't teach them in high school. Calculators have nothing to do with it.
Thank god we still teach quadratic equations, complex numbers, hyperbolic trig functions, and geometric constructions though. I don't know what would become of the world if most people didn't understand those things when we set them loose in the world.
lokar · 8h ago
So we just give up on direct reading comprehension and critical thinking?
Grading a stack of blue books is a "just kill me now" brutal experience. A majority of cognitive effort is just finding the page for the next problem to grade, with finding the answer another delay; any programming language with this kind of access latency would stay a "one bus" language. So of course professors rely on disinterested grad students to help grade. They'll make one pass through the stack, getting the hang of the problem and refining the point system about twenty blue books in, but never going back.
With stapled exams one problem per page one can instead sort into piles for scores 0-6 (if you think your workplace is petty, try to imagine a sincere conversation about whether an answer is worth 14 or 15 points out of 20), and it's easy to review piles.
When I had 200 linear algebra exams to grade at once, I'd scan everything, and use my own software to mark and bin one question at a time, making review passes a pleasure. I could grade a 10 question final in one intense sitting, with far more confidence in the results than team grading ever gave me.
First, its interesting to see a situation where a new technology INCREASED the cost of something.
Second, this is alluding to one of the dimensions along which that cost is going to be born, staffing. If its too many papers to grade, then more staff are needed to grade them.We're probably not going to get that, though, and instead the cost increases, the funding stays the same, so the quality will decline to compensate.
So it goes...
Also, second stapled exams with prescribed spaces for answers. So much time wasted just looking for where the answer is.
It's long been known that a longer essay answer is more likely to get a higher grade. And yeah, having been a student and a stressed out grad student, after about the 20th exam, only length is the real signal of grade.
Other comments point out that with the prices of tuition these days, students should be expecting a lot higher quality of feedback (grading) than what they are getting at any random R1.
It really does seem that the University system (as opposed to the college-esque system) is broken and that the additional AI fears are just another log on the already collapsed bridge. We're getting over the wrong thing.
It’s still a hard problem though. If the students have the laptops outside of the testing site, they can load cheating materials on them, or use them to smuggle questions and answers out if the test is used in multiple class sections. You realistically will not lock down a laptop students can take home sufficiently that some people won’t tamper with it.
Otherwise you have to have enough laptops to get each student a wiped and working machine for every test, even with lots of tests going on. And students need to be able to plug them in unless the batteries are rigorously tested and charged, but not every classroom has enough outlets. And you need to shuttle the laptops around campus.
Then you need a way to get the student work off the laptops. You probably want the student to bring the laptop up when done and plug in a USB printer. Anything else, like removable media, and you have to worry about data loss or corruption, including deliberate manipulation by students not doing well on the exam, and students claiming what got graded wasn’t what they meant to hand in. And you still have to worry about students finding a way to deliberately brick their laptops and the inevitable paper jams and other hardware failures, especially an issue when students need to leave on time to get to another class.
So you need systems that are cheap but reliable, tamper-resistant but easy to diagnose and maintain, robust against accidental data loss but easy to reliably erase, able to export data and install security updates without letting students surreptitiously input data, and heavily locked down while easy to use for students with a wide variety of backgrounds, training, and physical abilities.
The rack charges the laptops, streamlines distributing/collecting tests, prevents tampering, and reports defects. A professor uploads the test to their LMS and specifies the exam time-frame and location. Before the exam, the LMS transfers the test to the correct rack, which saves it to the laptops. After the exam, the rack loads the student responses and transfers them to the LMS for the correct class. Each laptop has a light under it whose color indicates whether it's in standby, waiting to be distributed (shortly before exam start), has a submitted test, or is malfunctioning (the rack periodically pings laptops in storage to verify they still work). Multiple professors can queue exams to the same location, in different time-frames, and the LMS and rack know when to prepare each test on the laptops.
The hard part is to implement this well. The rack's hardware and software must be reliable; nonetheless it will fail (hardware breaks), and exams get moved and rescheduled, so there must be a way to transfer tests to another classroom and time-slot. The laptops can have slightly less reliability, since there are backups; but failures can't be clustered, and if a laptop is transferred from another rack during an exam, it should download the destination rack's test and become active (so if one rack's backup laptops run out, laptops can be borrowed from other classrooms). The laptops must periodically backup in-progress tests using wifi or bluetooth, and if a laptop breaks during the exam, the student can resume their progress on another. Tests must be downloaded well before exam time, in case there are problems (laptops should be able to store and queue multiple tests). Laptops must handle exams that start late (up to the official exam end time) and end late (including overlapping the next exam's start time, in which case the next exam is loaded when the laptop is put back into the rack). The rack must absolutely not indicate that a laptop is ready with an exam if it's not; and (since tests are downloaded before exam time) nothing unique should happen at exam start time (to reduce last-minute surprises, and let a professor who isn't convinced take out a laptop and check that it's functional 15 minutes before). Last but certainly not least, the UI to submit tests and grade responses should be intuitive, simple (not overwhelming) yet powerful (can handle edge-cases like rescheduled tests, randomized tests, accommodations; or be extensible enough to support these features).
Despite all the above requirements I actually think such a product is feasible. Unfortunately, with firsthand experience of typical EdTech and academic bureaucracy (and I'm not a professor so I don't know the worst of it), I'm skeptical it would be adequate quality unless the company designing it is uniquely motivated and capable.
I've never seen anyone attempt blue books for anything else.
I guess what I'm asking is, how did AI shift the status quo for in class exams over, say, Safari?
A common mode I have seen is phone in lap, front-facing camera ingests an exam page hung over the edge of the desk. Student then flips the page and looks down for the answer.
Administrations won't allow it because they just don't care enough. It's a pain dealing with complaining parents and students.
In any case, cheating has existed since forever. There is nothing much new about cheating in in-class exams now with AI than before without.
AI is transformative here, in toto, in the total effect on cheating, because its the first time you can feasibly "transfer" the question in with a couple muscle-memory taps. I'm no expert, but I assume there's a substantive difference between IDing someone doing 200 thumb taps for ~40 word question versus 2.
(part I was missing personally was that they can easily have a second phone. same principle as my little bathroom-break-cheatsheet in 2005 - can't find what's undeclared, they're going to be averse to patting kids down)
Given the absolutely wild increases in tuition, administrations should have massive resources to bring to bear to solving this and other problems.
The reasons they don't feel more like excuses than legitimate explanations.
https://research.collegeboard.org/media/pdf/Trends-College-P...
Published tuition has gone up, but grant aid has matched it, making the net cost at 4-year public institutions to be flat, or even down slightly over the 10 year period. The same applies at private 4-year institutions, large increase in nameplate price, matched by large increase in grant aid, actual net tuition flat.
Expenditure data also show that they are not spending significantly more. See the chart at the end of this page, which gives expenditures in real, per-student dollars. They are up, a little less than 20% over 10 years, but half of that is increases in hospital costs of their associated health care systems, which have nothing to do with tuition. The rest is a mix of various activities of universities, much of which are not tuition-funded.
https://nces.ed.gov/programs/digest/d23/tables/dt23_334.10.a...
When I started out (and the original Van Halen was still together), blue book exams were the norm in humanities classes. I've had narrow experience with American undergrad classes the past 25 years, so I don't have a feeling for how things have evolved.
Why replace a system that generally works well with one that introduces additional potential problems?
Online instruction / learning can work for some people, and that's good.
I don't understand how anyone ever thought that an online exam could be made secure. There's just no way to ensure that the person who registered for the course is the one taking the exam when you don't control anything about the hardware or location, when students have a wide variety of hardware that you must support, and any attempt at remove video monitoring of the exam immediately runs into scalability and privacy issues. Like, even if you're watching a video of the person taking the online exam, how do they prove that they didn't just hook up an extra keyboard, mouse and (mirrored) monitor for person #2 to take the exam for them while they do their best to type and/or mouse in a convincing way?
It also doesn't help that you periodically get students who will try to wheedle, whinge, and weasel their way into an online exam, but then bomb the in-person exam (it's so strange and totally unrelated that they really, really wanted to take an online exam instead of in-person!).
Ok, I'll stop ranting now :)
The gist of it is that I think someone willing to put a lot of work in could probably cheat using the strategies you suggest, but it would be a pain. During your checkin you have to send a selfie, photos of the front and back of your photo ID, and four photos of the space you've prepared. You can't have any writing tools, written materials, or anything else that looks like a computer or screen in the area, and the machine you're on has to be single-screen. If the pre-test greeter or the proctor aren't satisfied with what they see they can ask you (via text and voice chat) to show them the room in real time via your webcam and may ask you to make changes or move things around to provide evidence that something in the room is not hiding mechanisms used to cheat. From that point on, your webcam and mic are on and live streaming to the proctor for the duration of the test; they don't say anything about assistive technologies on their end but I assume they are using eye tracking to look for instances of eyes wandering offscreen for a protracted period of time. The test environment software effectively "takes over your PC" during the test and I would imagine is pretty effective at detecting alternate/multiple display outputs etc.
There are probably scalability issues, but privacy is not an issue from the perspective of the proctor - you are effectively surrendering it by agreeing to take the test online, you could have gone to a test center instead.
That being said, the whole experience had an impact on my generally optimistic view of human nature.
There's a whole system for this, it already works very well if people actually wanted to make online exams work. Of course it's not "social distancing" so it didn't help with covid.
When I took physics we had weekly 3.5 hour lab sections. That should be enough for most CS assignments.
And, as someone who got paid minimum wage to proctor tests in college, I couldn't keep a straight face at this:
> The most cutting-edge educational technology of the future might very well be a stripped-down computer lab, located in a welcoming campus library, where students can complete assignments, by hand or on machines free of access to AI and the internet, in the presence of caring human proctors.
I think the author's leaning heavily on vibes to do the convincing here.
I have no idea what you're trying to express in your comment, so who's using vibes?*
Were you triggered by the word "caring?" A waiter usually cares that the people they're serving have an enjoyable meal. It doesn't mean that they love them, it means that they think the work of feeding people is purposeful and honest (and theirs.)
-----
[*] It's certainly not in the words; I don't know what made you angry about "joy," I don't know why you think the author does not teach writing skills in "communications," I don't know why the fact that you went through multiple drafts in writing school papers is relevant or different than anyone else's experience. Maybe that's over now. Maybe I actually don't care if you use AI for your second and further drafts, if I know you can write a first draft.
Drafting and redrafting a cumulative course paper, as well as iteratively honing a thesis, is a writing skill.
I would argue it as important than demonstrating recall and interconnection of the material. It is being lost if long-term work on papers is being replaced with 3-hour blue-book essays.
That is why I thought it was relevant. That's it.
Our interview usually starts with them breathlessly reading from a script out of the corner of their eye. I'm ok with notes to make sure you hit some high points about yourself even in person. Nervousness shouldn't disqualify a talented person. But with the coding part I've gotten exasperated and started asking these senior candidates to share their screen and do a fizz buzz exercise live in a text editor in the first few minutes. If they struggle I politely end the interview on the 15.
One candidate cheated and it was interesting to watch. In the time I sent the message in Zoom and them sharing their screen, just a few seconds, they had either queried or LLM-ed it on their phone or another computer, had someone off screen or in the same room listening and sharing the answer on another monitor or something else. Whatever it was they turned their head slightly to the side, squinted a bit and typed the answer in Java. A few syncopated characters at a time. When asked what modulo was they didn't know and couldn't make any changes to it. It was wacky. In retrospect I think it was them reading the question out loud to an LLM.
I'm waiting for the candidate who has someone behind them with the same shirt on pretending to be their arms.
These are the absolute worst.
You're taking people out of their comfort zone (highly customized IDE like JetBrains / VSCode / Vim) which cause them to lose shortcuts and decently working intellisense. Yes, my TypeScript in my projects is configured in such a way that I get way more information from the compiler than the standard config. After all, you're testing my ability as a software engineer, not a code monkey, right?
In this very uncomfortable place there is no way of asking questions. Yes, sometimes stuff is ambiguous. I rather have someone who asks questions vs someone who guesses and gets it right.
The testing setup is horrible too. No feedback as to what part of the tests fail, just... fail.
No debugger. No way of adding log messages. When was the last time you've been in that situation at your workplace?
All under the pressure of time, and additional stress from the person that they really NEED a new job.
Oh, and when you use compiled languages, they're way slower than say TypeScript due to the compilation phase.
And then even when your score (comprised of x passed tests and y failed tests) is of passing grade there is a manager out there looking at how many times someone tabbed outside of the window/tab?
Where am I supposed to look up stuff? Do you know all of this information by heart: https://doc.rust-lang.org/std/collections/struct.BTreeMap.ht...
Which reminded me that one time I used a function recently stabilized, but the Rust version used was about 8 versions behind. With that slow compilation cycle.
/sigh.
Everyone says this over the years, even before AI, and I've never felt it made the slightest difference in how they rate me.
On the few occasions (3) I took one, I ended up "punching through" the paper every single time. I tend to write with high pressure and the paper quality is atrocious. Twice, the tares were so bad the book was partially de-bound.
On both occasions, when presented with a "torn/destroyed book" I had to show the proctor the "issues" and then, very carefully, hand-copy over everything into a new book in their presence--absolute PITA.
Now we'll see the reverse, with students arguing that they can't handwrite effectively and need more time or whatever in order for exams to be fair. Hopefully handwritten exams will become the norm from grade school onward, so that this complaint will be rendered moot.
Currently, it's been a place for acquiring skills but also a sorting mechanism for us to know who the "best" are... I think we've put too much focus on the sorting mechanism aspect, enticing many to cheat without thinking about the fact that in doing so they shortchange themselves of actual skills.
I feel like some of the language here ("securing assessments in response to AI") really feels like they're worried more about sorting than the fact that the kids won't be developing critical thinking skills if they skip that step.
Maybe we can have
Why not open book + AI use exams, because that's what students will have in their careers?
We know, because we taught computers how to do both. The first long multiplication algorithm was written for the Colossus about 10 minutes after they got it working.
The first computer algebra system that could manage variable substitution had to wait for Lisp to be invented 10 years later.
https://www.sigcis.org/files/Haigh%20-%20Colossus%20and%20th...
The limitation seems to have been physical rather than logical.
Tools allow traversal of poorly understood, but recognized, subskills in a way that will make one effective in their job. An understanding of the entire stack of knowledge for every skill needed is an academic requirement born out of a lack of real world employment experience. For example, I don't need to know how LLMs work to use them effectively in my job or hobby.
We should stop spending so much time teaching kids crap that will ONLY satisfy tests and teachers but has a much reduced usefulness once they leave school.
I never need to "fall back" to the principles of multiplication. Multiplying by the 1s column, then the 10s, then the 100s feels more like a mental math trick (like the digits of multiples of 9 adding to 9) than a real foundational concept.
Oxford and Cambridge have a "tutorial" system that is a lot closer to what I would choose in an ideal world. You write an essay at home, over the course of a week, but then you have to read it to your professor, one on one, and they interrupt you as you go, asking clarifying questions, giving suggestions, etc. (This at least is how it worked for history tutorials when I was a visiting student at an Oxford college back in 2004-5 - not sure if it's still like that). It was by far the best education I ever had because you could get realtime expert feedback on your writing in an iterative process. And it is basically AI proof, because the moment they start getting quizzed on their thinking behind a sentence or claim in an essay, anyone who used ChatGPT to write it for them will be outed.
If they are trade schools, yes teach React and Node using LLMs (or whatever the enabling tools of the day are) and get on with it.
And the library, and inter-library loan (in my case), and talking to a professor with a draft...
And it did teach and evaluate skills I’ve used me entire career.
Because I was demonstrating that I understood the material intrinsically, not just knew how to use tools to answer it.
Making them open book + AI would just mean you need “larger” questions to be as effective a test, so you’re adding work for the graders for basically no reason.
For that, the student must have internalized certain concepts, ideas, connections. This is what has to be tested in a connectivity-free environment.
Faking intelligence with AI only works in an online-exclusive modality, and there’s a lot of real world circumstances where being able to speak, reason, and interpret on the fly without resorting to a handheld teleprompter is necessary if you want to be viewed positively. I think a lot of people are going to be enraged when they discover that dependency on AI is unattractive once AI is universally accessible. “But I benefited from that advantage! How dare they hold that against me!”
I get the same "you won't always have a calculator with you" vibes from 90s teachers chiding you to show your work when I hear people say stuff like this.
It's more likely I will not have paper and writing implements than not having a calculator.
Besides, most people have room for fast arithmetic or integrals; fast arithmetic would be more useful, but I'm not putting the time in to get it back.
Plus all about capability to actually retain whatever you ask from the model...
Challenge accepted. One possible solution: https://github.com/RonSijm/ButtFish
Thank god we still teach quadratic equations, complex numbers, hyperbolic trig functions, and geometric constructions though. I don't know what would become of the world if most people didn't understand those things when we set them loose in the world.