?

Log in

No account? Create an account
 
 
24 August 2011 @ 09:24 am
"couldn't retrieve anum for entry" error when exporting journals/comments  
Greetings fellow permanent account holders! If you have a moment, I wonder if you'd mind helping me troubleshoot something? I've been running into a really frustrating LJ bug, and I know that many permies also value the ability to back up their LJ content so I thought this might be a good place to ask.

Recently, any attempt that I make to export my entries and comments from Livejournal have failed. It's not like I get some content with bits missing here and there, rather I get no content whatsoever. Instead, whatever software I'm using immediately throws the error:

couldn't retrieve anum for entry at /home/lj/cgi-bin/ljprotocol.pl line 3952

Having read through LJ documentation, I understand what an "anum" is, but in sort of a catch 22 situation I am unable to verify if any of my entries are actually missing an anum in their database row because, well, I can't export my data with that kind of granularity...so there's no way for me to know if this error is due to a data corruption problem with my journal and a couple others, or if it's just a bug that hits LJ users on a more widespread level.

I don't want to inconvenience anyone, but if you already happen to use a backup/export tool (like ljdump, ljarchive, or export tools for dreamwidth, wordpress.com, etc) would you mind trying to sync/backup your journal real quick and just let me know whether you're getting this same error? Please note that Xjournal and the LJ export page still use the older, "flat" method and not XML-RPC (that's why they don't back up comments and lack other functionality).

If you find you are consistently getting this error like me, please submit a support ticket, similar to this one. Also, if you do, please let me know the URL to your ticket. I am linking to all related tickets in my own ticket's comments, in an attempt to help the developers who are working this bug. TIA!
 
 
 
Jessicajojobear99 on August 25th, 2011 12:59 am (UTC)
did you even try googling this? ;-) I googled "anum livejournal" and came up with http://www.livejournal.com/doc/server/appx.glossary.html which explains what anum means, which essentially is a random number for spam prevention.

you can also view the code for ljprotocol.pl online at:
http://code.livejournal.org/trac/livejournal/browser/trunk/cgi-bin/ljprotocol.pl
but unless you're a techie it won't mean much to you....

a better place to ask about what that error means would be lj_dev (though that has been asked before with no clear answer) as that's where the people who might understand that code used to hang out. But a support request might be more useful if you want to actually resolve this problem. My first guess would be there's some entry in your journal its hiccoughing on and figuring out whether the tool's broken or one of your journal entries is broken. I don't see this on the list of known issues (see http://www.livejournal.com/support/) so it could just be your journal ;-)
Jessicajojobear99 on August 25th, 2011 01:00 am (UTC)
apologies, I misread, you understand anum ;-) ignore half of what i just said. I'd still say go straight to filing a support request.
Quamzin Kravsherakhyron on August 25th, 2011 01:06 am (UTC)
In my post above, in addition to mentioning that I know what an "anum" is, I also mention support tickets and provide a link to the one I've already made. In case you're interested, here's a list of similar tickets I've culled together so far:

http://www.livejournal.com/support/see_request.bml?id=1364544
http://www.livejournal.com/support/see_request.bml?id=1366482
http://www.livejournal.com/support/see_request.bml?id=1365486
http://www.livejournal.com/support/see_request.bml?id=1366392
http://www.livejournal.com/support/see_request.bml?id=1367712
http://www.livejournal.com/support/see_request.bml?id=1367286
http://www.livejournal.com/support/see_request.bml?id=1367256
http://www.livejournal.com/support/see_request.bml?id=1365248

It's getting pretty frustrating. I think it's pretty safe to presume at this point that nobody can currently backup a journal using XML-RPC. I just thought this would be a good community in which to enlist some more "testers" beyond my circle of friends to verify that fact.
Jessicajojobear99 on August 25th, 2011 09:18 pm (UTC)
Yeah... part of the trouble is that when SixApart sold LJ, most of the employees who wrote most of the (code) infastructure and previously actively developed LJ stayed with SixApart instead (and personally, I don't blame them for that), causing a mass exodus of developers, and an almost complete turnover in employees for LJ. And around the same timeframe time more and more of LJ's codebase stopped being part of the open-source project and more of it domain specific/proprietary "enhancements" which further alienated some of the non-employee devs, and the whole lj dev community was essentially abandoned over time, and very few of the people who know the code well are still willing and able to commit bug fixes.

And none of that might all matter, except when you submit your support request, there's kind of a bare-bones staff when it comes to people qualified to look at the necessary DB fields, identify the problem, and rectify it...and compared the DDOS attacks recently, somehow exporting journal entries is probably pretty low on someone's priority queue. Unless you know the right people, or happen to have your own experimental/dev build from the open source code base, its probably hard to really troubleshoot yourself... Historically, I know there've been issues in the past with things like when they did the great unicode conversion ("upgrade"), and there were certain characters or entry corruption that would make the export tools barf horribly, so it would not in the least surprise me if there is something anomolous about the entry(ies) that cause the request to die. It is almost certainly a better solution to skip and continue if a particular entry is hosed, unless that is intentional to prevent some particular kind of exploit.
Quamzin Kravsherakhyron on August 25th, 2011 09:31 pm (UTC)
I understand the history, trust me...I imagine it's probably the people who "understand" the state of LJ in that way who use backup/export tools the most actually.

This code change between lines 3913 and 3915, where it appears "anum" has been added to a query which previously did not request it, may be involved (thanks to user av8rmike for the reminder).

http://code.livejournal.org/trac/livejournal/changeset/19669/trunk/cgi-bin/ljprotocol.pl

Now what I don't know is why anum was added to that query and the logic which follows it, but obviously what demanding it has done is borked functionality for anyone who (for whatever reason) has a single entry anywhere in an entire journal which may have been somehow committed to the table it lives in without an anum being generated (they're random).

What grates on me is, as you mentioned, this isn't being handled in what I would consider a reasonable fashion. If you're going to demand something that the query and handling logic didn't demand before, then you better think about the ramifications and handle those. If you're going to punt completely, for heaven's sake return the current ID so someone can fix the damn entry (by removing and reposting it, if all else fails). What might be even better would be just to effect a simple repair and generate an anum, if that's not deemed too costly (should it be?).

I wish, I wish, I wish there was SOME way for me to know what entry(s) might be missing one in my journal! I'd fix them lickety split and just move on. Waiting in limbo is painful. It's my content, and I want it.

*grrr*
Ron Newmanron_newman on August 25th, 2011 03:38 am (UTC)
I'm having no problems using ljdump on davis_square (where I'm one of the maintainers)
Quamzin Kravsherakhyron on August 25th, 2011 03:47 am (UTC)
That's interesting, I haven't tested any tools out on a community. I've just been trying to back up my journal. Do you mind trying to run ljdump on your own journal real quick just for kicks? You don't have to let it drag on or anything, if you get even one entry processed it will be much further than I can get (the missing anum error is triggered with my journal immediately).

If this issue is only effecting certain journals, I wonder what the discriminating factor is and why it's started recently?
Ron Newmanron_newman on August 25th, 2011 03:50 am (UTC)
It backs up my journal too, but I don't have much there.
Quamzin Kravsherakhyron on August 25th, 2011 03:55 am (UTC)
Fascinating. One theory that some folks have advanced is that the error is in fact "real," which is to say perhaps there isn't a bug in ljprotocol, but rather that problematic entries are to blame (which have been somehow saved to the database without an anum value, by client mishaps or LJ screwups or something). The fact that you can back up one journal while others can't certainly seems to lean in that direction...something journal-specific.

Of course the problem is that LJ gives us no access to see whether our data is messed up, and there appears to be no way to contact a support person directly to have a dialogue about which entries in one's journal might or might not have weird data in them (gawd, at this point if I could narrow the problem down to just an entry or two I'd freakin' delete them!).

I wish there was something more I could do about this, I feel like my hands are tied all of a sudden.
Fezziwig Fandango: Kat03bandicoot on August 25th, 2011 04:11 am (UTC)
Given the sudden rash of problems occurring about the same time leads me to point at LJ (probably XML-RPC) rather than the individual third-party apps. For example, my LJArchive errors don't mention "annum", but talk about lack of server response.

One thing I haven't seen mentioned is the question of what LJ did in response to the recent DDoS attacks - would any of those measures also have broken their API? Especially since I'm sure any measures they implemented were done in great haste.

Oh - one more thing. I have no problem importing entries, while one friend can't import those either. I just can't get comments.

I dunno...

Edited at 2011-08-25 04:12 am (UTC)
Quamzin Kravsherakhyron on August 25th, 2011 04:14 am (UTC)
For the journals this effects, it happens using absolutely any tool that talks to the XML-RPC interface (as opposed to the old "flat" method). It's definitely on LJ's side of things, there's no question about that. The way that XML-RPC works, these errors are being reported back to the backup/export tools from LJ, basically saying "hey I tried to handle the contents of this journal but I couldn't find what I was looking for and I'm punting now, sorry but I don't care to tell you what row I was on."

I guess the million dollar question is, what exactly changed to cause this to start happening?
Mister Baltimore: home sweet gotoav8rmike on August 25th, 2011 01:37 pm (UTC)
It's just a shot in the dark, but I looked through the code base and saw this change near line 3952. If "anum" didn't exist in the log2 table for a record (entry), the script might throw an error like people are seeing.
Quamzin Kravsherakhyron on August 25th, 2011 08:18 pm (UTC)
That exact diff was one of the first things I found when I started investigating the error (back when I didn't even know what an "anum" was). It seems very likely to me that the source of the error is related to it, but what remains to be seen I guess is why the case of a "missing" anum isn't being handled better (after all, it's trivial to just go ahead and add one to a row that didn't get one for some reason, and then go on about your business). It seems like a better way to handle the case would be to just repair what's wrong with the database row, then proceed to return the requested results to the client. Maybe I'm missing something.

Edited at 2011-08-25 08:18 pm (UTC)
(Deleted comment)
Quamzin Kravsherakhyron on August 25th, 2011 03:59 am (UTC)
Thanks for the feedback, so it's definitely journal-specific then. Man, I wish there was some magical tool I could run to find out which entries in my journal might have messed up data. This is maddening, oh how I wish I'd just backed up my entire journal with comments a few weeks ago before people started reporting this happening.
oh gallant piglet,aizome on August 25th, 2011 07:25 am (UTC)
Not sure if it's at all related, but I attempted a full back up at dreamwidth last night, but entries and comments failed after about fifteen minutes. Icons/profile info/etc. was successful, but not the journal content.

I'll try it again now, but if it fails again, I'll go ahead with submitting a support ticket.
Em: Weekdays // Fridayemploding on August 25th, 2011 10:42 am (UTC)
I've been considering doing a backup for some time now, and this seems like a good chance to do it.

Am running ljdump now, will let you know how it goes.

It's been going for a few minutes and seems to be going ok. So if you're getting an immediate error, it's not replicating for me.
Non-Recovering WoW Addictheinous_bitca on August 25th, 2011 12:59 pm (UTC)
Tried using ljdump since I'm on a Mac at work.

xmlrpclib.Fault: < Fault Server: "couldn't retrieve anum for entry at /home/lj/cgi-bin/ljprotocol.pl line 3952\n" >

Tried it on my much smaller second journal, and it went through fine, no problems.

Edited at 2011-08-25 12:59 pm (UTC)
Quamzin Kravsherakhyron on August 27th, 2011 08:26 am (UTC)
an update from support
I got an e-mail reply from support due to all the comments I've been adding on to my ticket as this has been developing. I figured I'd paste it here, while there's no solid info we can at least all take some comfort in knowing attention is officially being paid and some sleuthing has been done. Fingers crossed!

Thank you for your extra information. I have passed this information on to developers. (This problem has actually been happening since before that specific change happened, but this tip allowed us to find an earlier change that probably did trigger this influx of problems at this time.) I am sorry that we don't have a timeline on when this will be resolved, but continuing to watch lj_releases will get you the most updated information on when bugfixes are live.
Magic Magpiequirrc on August 30th, 2011 08:02 am (UTC)
Did you try Semagic? Links/Synchronization. It does not use xml-rpc.
Quamzin Kravsherakhyron on August 30th, 2011 02:48 pm (UTC)
I'm aware that Semagic can download comments as well as entries (as I recall, it does so by spidering LJ, which isn't the most server-friendly method but at this point I'm too frustrated to care). However I'm not sure whether the XML it produces will actually be useful to me (able to be imported into another blog, like Wordpress).

I decided to give it a shot anyway, and just spent an hour building an up-to-date Windows virtual machine to run it in...and guess what?

Error 500!

Sorry the service is currently unavailable due to technical issues. The page you requested is not available. Frank has been chewing on the wires again.


It's simple, LJ can sense that I want to get out of here and hates me. *sigh*
Magic Magpiequirrc on August 30th, 2011 04:33 pm (UTC)
Semagic now does it more server friendly than any other program. 500 error is for all users, it is turned on and off several times a day.
Quamzin Kravsherakhyron on August 30th, 2011 05:18 pm (UTC)
Thanks, I see lots of 500 error tickets and complaints now that I look around. Hopefully I'll be able to try this out later...I'm firing up the VM and attempting every 15-30min but so far no luck (the error is instantaneous each time).

I suppose since Semagic implements synchronization on its own, and therefore never respiders an entire journal or anything, it's probably at least as "friendly" as the XML-RPC interface is supposed to be. Ultimately I guess it matters little in my case, because I'm trying to take a complete backup from scratch.
Quamzin Kravsherakhyron on August 31st, 2011 02:35 pm (UTC)
I must be missing the times you're "allowed" to back up? It's been more than 24 hours since I started trying and every single attempt, I get the 500 immediately (not a single entry or comment comes in, not one). Do they throttle access via clients like Semagic or something? Is something weird going on with my specific account? I wish I knew more about the mysteries of LJ.
Magic Magpiequirrc on August 31st, 2011 03:07 pm (UTC)
I have no idea what is going on, maybe they count server load for particular cluster or something. I just tried and it works. Before one time I tried for 2 journals and one worked and another did not.
angharad governal: cup o' teaangharad_gov on September 3rd, 2011 12:26 am (UTC)
just adding a link to my own filed support request if you're still collecting links. having issues w/ LJ Archive as well:

http://www.livejournal.com/support/see_request.bml?id=1373088
lisalisabella on November 1st, 2011 09:57 pm (UTC)
Any resolution to this? I can no longer run LJ Comment Stats and was wondering if there's a connection there.