Wikilivres:Community Portal/en/Archive 2013

Starting import of URAA deletions on Commons

I've developed a private bot that can import files to Wikilivres from Wikimedia Commons that have already been deleted. I'm applying it to any files listed in deletion requests that are PD in Canada, with a special emphasis on URAA-related deletions. You can see my uploads at Special:Contributions/Transfer from Wikimedia Commons bot and some earlier ones at Special:Contributions/Dcoetzee, and see affected deletion requests at "Category:Deleted files transferred to Wikilivres" at Commons. Many templates are missing here, which I'm gradually importing as needed, but I'll need help with the Wikimedia message extensions (see below). Dcoetzee 16:24, 12 January 2013 (EST)

Thanks a lot for that. I would give it the bot flag unless someone objects. Yann 10:33, 13 January 2013 (EST)

The biggest problem with dumping this material here is that we are not equipped to handle and categorize so many media files. Also it is not enough to simply add templates from Commons without documentation, or to justify changes on the sole basis of their conformity to Commons. Changes need to be explained in more detail than that. Eclecticology 21:45, 13 January 2013 (EST)

What kind of information would you like to be added? I have given it the bot flag. Regards, Yann 01:06, 14 January 2013 (EST)

The imported templates should have full proper documentation telling how they work, and how they are used. We already have a large array of templates that make their usage difficult to trace or change. As a simple example there was a change to Template:en to conform to Commons usage without any explanation about what the change would do.

As for the pictures themselves, we need to remember that this project was a Wikisource spinoff, not a Commons spinoff. No system of categories to handle pictures has ever been developed here. Some of the pictures added have no category at all; others have categories so obscure that they cannot possibly help in finding the pictures. Who is going to develop a functioning category system? Eclecticology 05:53, 14 January 2013 (EST)

As noted on my talk page, I'll update all the template documentation (Template:En and the other lang templates are already updated), and assist in the development of the category system, and have recruited other people from Commons to help too. I will not be uploading images at such a fast rate in the future. Dcoetzee 08:33, 14 January 2013 (EST)

Missing Wikimedia messages

In order for my imported description pages from Commons to render correctly (see e.g. File:HC_Baruch_Ponstijn,_when_young,_by_Leo_Gestel.jpg), this wiki needs the extensions WikimediaLicenseTexts and WikimediaMessages. Please install them. This will also fix Template:LangSwitch, which uses "{{int:Lang}}" to pick the language to display, but is not currently working. Once these are installed, I can deal with the remaining templates by manual import. Thanks! Dcoetzee 14:48, 12 January 2013 (EST)

Issue of PD-Art images

A number of the images I'm uploading are PD-Art (simple reproductions of public domain works where we don't have a license from the photographer). Since Template:PD-Art exists I'm assuming they're accepted here, but according to Commons:Reuse_of_PD-Art_photographs#Canada these files may or may not be in the public domain in category (case law is inconclusive). So we should explicitly discuss and agree on a PD-Art policy. Dcoetzee 16:36, 12 January 2013 (EST)

I think we should accept here at least what is accepted on Commons. So any reproductions of paintings are OK, if the painting is in the public domain in Canada. Yann 10:32, 13 January 2013 (EST)

I have no problem with this. While I haven’t reviewed the Canadian cases on this point, this seems a fair position. Eclecticology 19:59, 13 January 2013 (EST)

Unable to upload image files

I can't upload this file (Egger-Lienz - Andreas Hofer - 1923.jpeg, 11 MB) at all. I tried renaming it to a jpg extension, tried uploading it from a computer with lots of upload bandwidth, but all to no avail. I just get back the same upload page with no error message. Upload by URL doesn't work either. I've uploaded lots of other images with no problem. I'm not sure what's wrong but I'm guessing there's some misconfiguration leading to a filesize limit of ~~10 MB~~ 8 MB or so. Dcoetzee 17:33, 12 January 2013 (EST) Update: I had the same issue with re-uploading the following files from Commons:

File:Otto Mueller - Selbstbildnis mit Pentagramm - ca1924.jpeg (14 MB)
File:Bouquet Portrait de Le Fauconnier.tif (8.1 MB)
Sergei Rachmaninoff - Symphonic dances (1).ogg (11 MB)
Sergei Rachmaninoff - Symphonic dances (2).ogg (9.3 MB)
Sergei Rachmaninoff - Symphonic dances (3).ogg (9.4 MB)
Christian Rohlfs - Erfurt (Dom und Severikirche).jpeg (8.0 MB)
Rohlfs - Aus Dinkelsbühl, 1924.jpeg (8.1 MB)
Christian Rohlfs - Porträt einer Frau mit Hut.jpeg (12.3)
Rohlfs - Engel, der Licht in die Gräber trägt, ca1925.jpeg (8.6 MB)
Rohlfs - Kloster Andechs.jpeg (10.6 MB)
Rohlfs - Tänzerpaar, 1928.jpeg (10.4 MB)
Rohlfs- Spiel am Strand, 1928.jpeg (8.7 MB)
Rohlfs - Austreibung aus dem Paradies, 1933.jpeg (8.9 MB)
Christian Rohlfs Buchenwald.jpg (17.5 MB)

I had no trouble uploading smaller TIF and OGG files (like File:Bouquet Pégase.tif at 2.8 MB and File:Sergei_Rachmaninoff_-_Concerto_1_in_F_minor,_2nd_movement.ogg at 6.5 MB) reinforcing that it's a file size/configuration issue and the threshold is between 6.5 and 8.0 MB. Dcoetzee 22:08, 12 January 2013 (EST)

Still need help with this. If there are no developers around, I could fix it myself if I had shell access to the server, I would just need the credentials/key. I run several MediaWiki servers and there is clear documentation of how to deal with this issue at Manual:Configuring file uploads: Set maximum size for file uploads. If the site does not wish to increase its limit above 8 MB for whatever reason, please let me know and I'll find a way to work around it. Dcoetzee 23:03, 2 February 2013 (EST)

I must confess that my technical abilities about server issues are non-existent, and don’t have the credentials. I wouldn’t know what to do with them if I did. Alan (awalker@wikimedia.ca) is the one in position. Try contacting him. Eclecticology 00:58, 3 February 2013 (EST)

Thank you! I'll do so. Dcoetzee 02:31, 6 February 2013 (EST)

Michelin maps

There is a discussion going on for the file [1] It wil problably result in a URAA deletion. Can this be transferred to wikilivres? I have other uploads wich are not flagged but probably follow the same delete proces (Category:Michelin maps). Smiley.toerist 08:58, 16 January 2013 (EST)

From a copyright perspective there is no problem as this map is more than 50 years old. My concern would have more to do with the import of 11 Megabite files. Eclecticology 15:20, 16 January 2013 (EST)

I could reduce the size. I'm used to the commons where size is not really a problem. Smiley.toerist 06:53, 28 January 2013 (EST)

As noted above, the current limit seems to be 8 MB. I really hope this can be increased as I have many files larger than this to import. Dcoetzee 23:07, 2 February 2013 (EST)

djvu still reports no such file

It appears that people have experienced the "no such file" error with the mediawiki:Extension:Proofread_Page for djvu files, see examples at Index:Chesterton - The Coloured Lands.djvu and Index:Der weiße Maulwurf.djvu.

Today I uploaded a new djvu file (created with DjVu Solo), but despite all my pokings I could not get the extension to work as it should, and the Index pages report "Database error" messages with the pagelist tag, see Index talk:Chesterton - The Coloured Lands - Pictures From The Paint-Box.djvu.

A similar error was reported at Proofread error: "no such file" but there was no answer.

I am requesting some help at mediawiki here. -84user 18:03, 19 January 2013 (EST)

Update so far the sole response at Extension talk:Proofread Page#no such file at wikilivres was to check the DjVu configuration on wikilivre's server, linking to Manual:How to use DjVu with MediaWiki. -84user 19:18, 23 January 2013 (EST)

Hi, I get a look and I think too you have a configuration trouble, perhaps the extension is mis-configured but you have trouble before reaching the extension, get a look at

, only an icon is shown, and mediawiki don't get the size of the first page nor the number of page, this pinpoint a trouble in the installation of djvulibre tool or a mis-configuration of the mediawiki core, not a trouble in the extension. Phe 18:15, 30 January 2013 (EST)

À la recherche du temps perdu

I would like to upload the last three volumes of this work by Marcel Proust in the original French to Wikilivres. They are in five books with a combined size of less than 55 megabytes. They are all match and split jobs with good quality texts from the Bibliothèque Électronique de Québec. The third book also has 84 pages of validated texts. They are Gallimard texts which still have copyright in the U.S. until 2018, 2020 and 2022 and must be deleted from Commons until those dates. ResScholar 19:07, 26 January 2013 (EST)

Go ahead. Eclecticology 05:10, 27 January 2013 (EST)

Thank you. I will ask User:Dcoetzee to see if he can assist the migration. After all, this work, too, is a URAA restoration item. ResScholar 19:00, 27 January 2013 (EST)

Contributing as a Non-Canadian

Hi! Regarding contributions of Non-Canadian citizens to this project (who want to upload works which are still copyrighted in their country): Do you recommend them to contribute anonymously? Is it mentioned at any FAQ-page? --Question2013-05 14:46, 2 May 2013 (AKDT)

This is entirely up to you, and how aggressive your own country is about copyright enforcements. When I review what someone has contributed I always do so strictly according to Canadian law, and I am certainly willing to accept legal responsibility in Canada for what I review. It would take an order of a Canadian court before your identifying details would be revealed, It all comes down to your personal tolerance for risk Eclecticology 04:37, 4 May 2013 (AKDT)

Thanks

I want to extend a big thank you to users Electron, Zephyrus and Luciana for their continuing hard work fighting back against a persistent spam attack. New user Matma Rex has already been a blessing to the project by adding a spam filter and by finding and fixing the extreme slowness that this site has been experiencing. I have also asked him to look into upgrading the MediaWiki software. This may also lead to fixing some of the other problems that have been mentioned in other posts to this page. Eclecticology 11:09, 27 July 2013 (AKDT)

For what it's worth I would still like to participate as a developer. I know how to fix the problems I listed above but require a shell account on the server. There's also a new problem rendering thumbnails, see e.g. File:WLANL - kwispeltail - Mediterranee 2.jpg. Who should I contact? Thank you. Dcoetzee 10:07, 7 August 2013 (AKDT)

Well, I think that Eclecticology can help you. Btw. In my opinion the best way to get rid of the spam bot that creating new user actounts permamantly is to install a strong captcha filter on Sign up page, like is e.g. on Wikipedia. Electron ツ ➧☎ 00:36, 9 August 2013 (AKDT)

I'll look into all this when I get home from Wikimania. Eclecticology 04:22, 10 August 2013 (AKDT)

Spam bot

Ack with Electron, the first step towards reducing the creations of the spam bot accounts would be the installing of a strong captcha filter. On some days it is quite awful. -jkb- 10:43, 15 August 2013 (AKDT)

For today's spamming: see here -jkb- 06:00, 17 August 2013 (AKDT)

Btw. A good idea: before instaling a strong capcha filter to reduce creating new accounts, there can help, a new filter for AbuseFilter that stop creating accounts with numbers in the name. So far it can be edit only by beauroucrats -> Special:AbuseFilter, so I can't help. Some useful filters you can see e.g. on Wikipedia. Electron ツ ➧☎ 03:35, 19 August 2013 (AKDT)

hier I moved my edit from yesterday on the talk page of Eclecticology:

I think the abuse filter works well all the time; but notice, the filter works only when new users try to do something, that should be abandoned: the spam accounts, where spamming is added, are not new but "autoconfirmed" (the spam bot is waiting some days and start spamming later)
thus a strong captcha for the registration could be the first help
this would probably be necessary as the spam bot will try to change the IP (not very difficult, it can use open proxys as well, I guess)
if a new situation lieke yesterday accure again, we really need a bot with admin rights; yesterday a started to delete and to block the accounts, but the bot was hardly quicker with spamming as me with my measures. I stopped my action then.

/end of the move; let us discuss on one place/
-jkb- 07:03, 19 August 2013 (AKDT)

Thanks; following the threads was confusing me. Eclecticology 22:38, 19 August 2013 (AKDT)
To -jkb-:

Generally you are right. But there are no one tools that won the spam at ones; they can only help decrease spam in some percent - but percent plus percent can make a big sum. So the best solution is usually a combination of some tools.
My bot (User:Electron Bot) has admin rights already but I don't know the tools that can block new users automaticly (now I am using AWB and pywikipedia scripts and I am not a script programist). If you known a tool that can help I will try to use it...
Big trable require strong measures, usually. Maybe there is a need to stop let creating new user accounts for IP users temporarily? It is a radical step but during last time, no one of new users is a human. It can be done by adding new fiter to AbuseFilter that can also inform new users that if they want to have account here it can be creating for them by the admin. Of cource, the filter can be switched off as soon as we have other tools that can handle the spam, e.g. a strong captcha. Electron ツ ➧☎ 23:33, 19 August 2013 (AKDT)

I never had a bot so I don't know how they work, i.e. what a sort of a table they need to reed what is to do. When you see the RC every day so you can copy the new users somewhere, might be you can delete some of them whos name is Andrea Smith and not Xsz871ght, and probably the bot could do the job.
Anyway, all the bot created accounts are now waitig for some 3 or 4 days befor starting with the spam, that means that the abuse filter #1 works.
I was very skeptical if we sould need another spam as the spammed texts are pretty different. But since the last night there were some 15 accounts that spammed with the text like "Windows 7 Key". Here we could arrange a simple small filter just like I did on the oldwikisource.
To the last point of Electron: well, when somebody would suppose that on a de- or fr- or en-wiki, it would be a nogo. Some short time restriction in the registration could be done, might be, as there are not such many new users here. If a new user comes here and if he see the recent changes so it is not a good renomée for us. And if we would post a sitenotice or something like that explaining why we do so, why not.
-jkb- 07:23, 25 August 2013 (AKDT)

Please, Uuxw4e7p, do excuse me: you contributed with only one contribution.

                05:38 . . Zephyrus (Talk | contribs | block) deleted "User:Lztblx049" (content was: "== IThat may be well and agreeable They /definition.html are very ..." (and the only contributor was "Uuxw4e7p"))
                05:37 . . Zephyrus (Talk | contribs | block) deleted "User:Sjxpip743" (content was: "== IThat may be well and agreeable They /definition.html are very ..." (and the only contributor was "Uuxw4e7p"))
                05:37 . . Zephyrus (Talk | contribs | block) deleted "User:Uuxl5d4w" (content was: "== IThat may be well and agreeable They /definition.html are very ..." (and the only contributor was "Uuxw4e7p"))
                05:36 . . Zephyrus (Talk | contribs | block) deleted "User:Uuxm8b8p" (content was: "== IThat may be well and agreeable They /definition.html are very ..." (and the only contributor was "Uuxw4e7p"))
                05:35 . . Zephyrus (Talk | contribs | block) deleted "User:Ir6d6g9r" (content was: "== IThat may be well and agreeable They /definition.html are very ..." (and the only contributor was "Uuxw4e7p"))

This contribution alone came from you, Uuxw4e7p:

05:35 . . Zephyrus (Talk | contribs | block) deleted "User:Uuxw4e7p" (content was: "== IThat may be well and agreeable Theyo we /definition.html are very ..." (and the only contributor was "Uuxw4e7p"))

Please, Uuxw4e7p, do forgive this unintentional error of mine. Kindest regards,

</il faut bien rire un peu>

--Zephyrus 22:20, 27 August 2013 (AKDT)

New filters of AbuseFiter can stop the spam but if nobody do nothing we have what we have... Electron ツ ➧☎ 03:52, 2 September 2013 (AKDT)

I will do what you tell me to do: you are the expert Electron. Were the deletings I was doing useful or were they hampering the bot's work? Would it be useful if I delete the users categorized spam by your bot or is it better to let the bot delete them? Is it useful that I block these users? Many many thanks for your help ! --Zephyrus 08:38, 2 September 2013 (AKDT)

I think that the most important thing (on this moment) is to stop creating new accounts. It can be done by adding new fiter to AbuseFiler. New account can be created only by admins on an user request. After stoping creating new accounts we should clean the mess done by the spam bot, gradually and think what to do next...

What about my bot: After instaling AbuseFiler I have problem with page deleting with my bot because of an error appearance: ApiErrorException in ApiEdit.CheckForErrors WikiFunctions.API.ApiErrorException: Bot API returned the following error: 'Exception Caught: Detected bug in an extension! Hook AbuseFilterHooks::onArticleDelete failed to return a value; should return true to continue hook processing or false to abort.' I am not familiar with API so I don't know what the problem is. I thing that this is a problem with AbuseFiler or its configuration because I try two bot tools (AWB and pywikipedia) and the both have problems with page deleting. So, on this time, I can only collect all new spam in Category:Spam for the future blocking and deleting. Of course it can be done manually but it is a hard work to do by hands... Electron ツ ➧☎ 04:37, 3 September 2013 (AKDT)

Your suggestion to stop creating new accounts looks the right thing to do now in my opinion too, but perhaps there are faces of the problem that I'm not aware of; so let EC decide, knowing what we think. I will go on blocking spammers by hand if we have no other solution left. --Zephyrus 10:46, 3 September 2013 (AKDT)

OK. But the decision should be taken quickly - the spam bot produces about 300-400 new pages every day... Electron ツ ➧☎ 01:16, 4 September 2013 (AKDT)

As I said above: yes, we should stop creating new accounts in fact immediately, then we should decide, how to create them in the future (strong captcha), and the we must 1) delete all those damned pages and 2) block all spam bot acconts. As there are hundreds and thousand of the, so I think manual deleting and blocking is not a very good idea. -jkb- 14:58, 4 September 2013 (AKDT)

P.S. The most spam edits in the last days contained the wording "?mod=space" etc. This could be used for a single abuse filter. -jkb- 15:40, 4 September 2013 (AKDT)

I think I figured out how to stop creation of new accounts. Somebody try it to see if it works. Eclecticology 16:31, 4 September 2013 (AKDT)

It is very easy to perform. The filter can be looks like this:

Conditions: action == "createaccount" & user_groups != "sysop"

Actions taken when matched:

Flag the edit in the abuse log

Prevent the user from performing the action in question

Block the user and/or IP address from editing

Of course the conditions can be more complex and depend of many other ruls, to precise it. But the spam bot has stopped, now. Electron ツ ➧☎ 23:36, 4 September 2013 (AKDT)

I found from the Media Wiki site that I could add "$wgGroupPermissions['*']['createaccount'] = false;" to the LocalSettings.php Eclecticology 00:47, 5 September 2013 (AKDT)

YES!! I don't know this as I don't need it on my local wiki at home... -jkb- 00:54, 5 September 2013 (AKDT)

Yes, it can be done there, as well. And will make result like the filter. Electron Bot 01:01, 5 September 2013 (AKDT)

Footer for IPs

In MediaWiki:Sp-contributions-footer-anon I created a small footer for anon users: if you see the special paqe of an IP for contributions like here, you will find on the bottom some usefull informations about it. -jkb- 10:07, 5 September 2013 (AKDT)

Great! Now I just need to find out how best to use these tools. Eclecticology 14:41, 5 September 2013 (AKDT)

Well, the Geo IP ist quite good. It doesn't show the city quite properly (but China is great enough...). Sometimes it is good to know the location, although not every Chinese IP is an open proxy (there are another tools to fix them). But when an edit from China, reverted (and user blocked) comes five minutes again from Ukraine, then you can assume... :-) -jkb- 02:20, 6 September 2013 (AKDT)

Blocking the open proxies of the bot

After Eclecticology succeeded to stop the creation of the bot accounts (and, pls, see if my sitenotice is OK), we have now the problem wirh the old accounts that are autoconfirmed and can edit new pages. We can have two strategies:

we can check the IPs and block the range block, up to now it was done indefinitely.
we can check the actual IP and block this single IP for quite a short time, I guess 1 month is enough.

This afternoon I made a test (see User:-jkb-/sbm) and I think I know how the bot works. He edits with 1 IP (normally OP, today from China), after this single IP is beeng blocked, he starts some time later with another IP - just fully another range (!!) und continues. It doesn't matter if you block the full range or the single IP only.

Blocking the ranges is a bit problematically: if you block a renge like 36.249.0.0/16 (with a 16 on the end), so you block 65536 IPs in the same time! Some points:

an open proxy is not an open proxy in the whole future - they come and they go
the spam bot has the same behaviouin the both cases - s. above
every time he cannot use the present IP he changes to quite another range

Might be blocking of the single IPs could be enough before we have a new filter / filters etc. -jkb- 10:09, 5 September 2013 (AKDT)

The site notice is just fine.

I have been blocking 0/16 ranges indefinitely without feeling guilty about it. :)

When you use checkuser on a user name it will show if he has been editing from more than one IP address. If I then plug each range xxx.xxx.xxx.0/16 and get the edits from that address it will show how many addresses have been used within that range. This includes accounts that were set up but never used; some were set up a month or two ago. I've limited blocking only single IPs to where that is the only IP used from that range.

At this moment there are 1148 items in Category:Spam, plus whatever spamsters have had their work deleted without being blocked, and the sleeper accounts that have never edited. Lots of work still! Eclecticology 15:27, 5 September 2013 (AKDT)

Well, now we have 27756 "users", most of them created by the spam bot... Electron ツ ➧☎ 02:14, 6 September 2013 (AKDT)

Blocking ranges indef: well, soon we have no newbies that could register here :-)... But: We hope we shall solve this problem soon - e.g. with a captcha registration. After this we schall not have such big problems. Thus I thonk we can use the Meta regulation saying, open proxies should be blocked max one year only (I normally use 6 months). When you find one or more OPs in the range xxx.xxx.xxx.0/16, it doesn't mean that all 65 thousands IPs are the same. And, as I said, the spam bot changes the ranges very soon after beeing blocked. -jkb- 02:30, 6 September 2013 (AKDT)

P.S. I have just asked somebody in deWP how to arrange a captcha. -jkb- 02:30, 6 September 2013 (AKDT)

Captcha

There are two sorts of captcha: the single one, where you must just add a number to another (like 4 + 7 = "?"), and the more difficult one ("Fancy Captcha")like in the wikies now (where you must write a word which is elaborated from letter pictures).

Simple captcha: the extension is on https://www.mediawiki.org/wiki/Extension:ConfirmEdit, incl. some manual and a download link; it must be downloaded, the .gr file must be zipped, and then the .tar file must be zipped, and the result must be saved in the folder ConfirmEdit ind the folder /extensions/ (in the mediawiki structure). Then in the LocalSettings.php two lines must be added:

require_once "$IP/extensions/ConfirmEdit/ConfirmEdit.php";

$wgCaptchaClass = 'SimpleCaptcha';

The captcha is activated now, but it is necessary to change the options

$wgCaptchaTriggers['edit'] = false; // Would check on every edit

$wgCaptchaTriggers['create'] = false; // Check on page creation.

$wgCaptchaTriggers['sendemail'] = false; // Special:Emailuser

$wgCaptchaTriggers['addurl'] = true; // Check on edits that add URLs

$wgCaptchaTriggers['createaccount'] = true; // Special:Userlogin&type=signup

$wgCaptchaTriggers['badlogin'] = true; // Special:Userlogin after failure

where in the line addurl you change it from true to false.

Alternatively you can use the extension https://www.mediawiki.org/wiki/Manual:$wgAccountCreationThrottle, which allows one syccount creation in 24 hours only. A manual should be there.

The FancyCaptcha is a bit comlicated as all the images must be uploaded (or generated) first to the server.

Thanks to User:Umherirrender from German Wiki.

Regards -jkb- 07:49, 8 September 2013 (AKDT)

P.S. SingleCaptche and the AccountCreationThrottle can be installed both. -jkb- 10:01, 8 September 2013 (AKDT)

Good! For now my priority is to get rid of these sleeping spam bot accounts before they come back to bite. The one block from a group of 4 IP addresses is responsible for nearly 10,000 accounts. I did try something before that tried to impose captchas, but it crashed the site. Once I'm finished with the sleepers, I can take a serious look at the above suggestions. Eclecticology 12:26, 8 September 2013 (AKDT)

Yes, it has not to be done in the next two hours :-)... Moreover, I've got some other suggestions just now, I will post it here tomorrow. -jkb- 13:46, 8 September 2013 (AKDT)

OK. Raymond from the dewiki wrote me some points as well. He say the simple captcha cannot be recommended seriously against spam bots. But I would say we could try it as the first and quick step just to open our registration soon. I would recommend to combine this simple captcha with the Throttle mode (see above) which allows one registration from a IP in 24 hours. If it doesn't help we can do something else, but this is probably not very difficult.

By the way: congratulations to all to this!!! Regards. -jkb- 08:54, 9 September 2013 (AKDT)

That sounds like a workable plan. Some of the accounts that I've been blocking were set up at the same time down to the second. Yes, great work by those who have been working through that spam category! Eclecticology 09:13, 9 September 2013 (AKDT)

OK. All known pages with spam were delated and blocked. Last pieces of them, that was omited by my bot I have found and deleted from this site -> Special:ActiveUsers. Of corse it is a good idea to block all of the new accounts created by the spam bot, but I think that it can be done gradually. Electron ツ ➧☎ 14:45, 9 September 2013 (AKDT)

It takes a while. I've been working through 222.247.36.189's 4000+ accounts. If I do too many in a day it screws up recent changes. Eclecticology 16:54, 9 September 2013 (AKDT)

Recent changes are awful anyway in last days. Wouldn't it be better to create a new account and to grant it the bot status? Then it can be hidden from the rc. -jkb- 01:01, 10 September 2013 (AKDT)

It's a good idea. A bot edition aren't seen on default on RC. Electron ツ ➧☎ 04:13, 10 September 2013 (AKDT)

True enough, but I think that we will be rid of these bad accounts before the bot is written. Eclecticology 10:15, 10 September 2013 (AKDT)

... "before the bot is written..." - I think you do not need to write any bot programm or something like that - it is enough if an account has the bot flag, then it is invisible in the rc, even if you make the edits manually. -jkb- 15:39, 10 September 2013 (AKDT)

OK. I'll try that with my secondary account. Eclecticology 16:51, 10 September 2013 (AKDT)

Great, it works! Eclecticology2 17:24, 10 September 2013 (AKDT)

Great, it works! :-) -jkb- 22:16, 10 September 2013 (AKDT)

After blocking some 10,000 spambot accounts, my mind is numb. Now I can take a closer look at captchas. Eclecticology 23:05, 13 September 2013 (AKDT)

I have implemented the simple captcha. Matma Rex had commented out the first line as "doesn't help any". The other line also had been commented out with a #, but possibly earlier by some other person. The captcha seems to work now. I have restored account creation, but will be keeping a very close watch on this. I will continue looking at the throttle option for an additional level of security. Eclecticology 19:56, 14 September 2013 (AKDT)

It doesn't look good, the bot is obviously able to register new accounts even if there is the sinple captcha. Just like I was warned. I'm not sure if the throttle would help a lot. In the last hours I was online and I blocked the IPs immediately, but after a time the bot found a new OP. I takes more time but he can do it. -jkb- 02:51, 15 September 2013 (AKDT)

P.S. And there seems to be something new - the user names are not as spam bot like (qprt456x) as in the past, now the bot uses "normal" names mostly. -jkb- 02:58, 15 September 2013 (AKDT)

Well, I analysed a bit the new accounts registered today. The results are on the page User:-jkb-/sbm in the thread 15.9.2013 (as the page contents many personal data and as nobody is invited to read it, I deleted the page; if somebody waqnts to read it, so klick on the last version and see the preview, if adding something so renew the page and delete it at the end). -jkb- 04:24, 15 September 2013 (AKDT)

I guess we have to stop the registration of new user again, and I do not think, the throttle (1 registration avery 24 hours only) would help, as the spam bot uses new OPs every time. Probably a more complicated capthca should be installed. -jkb- 08:29, 15 September 2013 (AKDT)

The human sounding names suggest human spammers. I have been looking at the history from some of these IP ranges, and blocking ranges when more than one address is used from that range. I'm also blocking users whom we missed when we were blocking thousands. The ones that are appearing now did not generate large numbers of accounts, and seldom more than one per day. I agree that the throttle won't help with these. A human spammer that only appears once a day is not likely to be bothered by answering a captcha of any kind. I'm wondering whether some kin of email address verification would work. Eclecticology 10:12, 15 September 2013 (AKDT)

Well, I think it is the question of programming - a bot can generate human sounding names as well, you just have to give him a list of names and surnamen (see: in wikipedia :-) )... What still is a question for me is the fact, that the accounts are obviously registered from more then one PC - see my subpage -jkb-/spm. Anyway, I would favorize a quick suspendig of the registration and a new try with a most complicated captcha. If we allow the registration then nothing changes, we have dozens and hundreds new spam accounts every day. Regards -jkb- 15:03, 15 September 2013 (AKDT)

results of the blocking

OK, making a very quick and not exact statistic (just going along 500 blocked by 500 blocked in the log) we have this result:

Electron Bot 1 500

Eclecticology 18 000

Eclecticology2 9 500

Luciana 1 500

-jkb- up to 500

so together some 31 000 blocked spam accounts. Not bad, indeed :-) -jkb- 00:57, 14 September 2013 (AKDT)

With Yann's, Poetlister's and mine, 815 more. What can I do now? --Zephyrus 02:14, 14 September 2013 (AKDT)

Oh, so sorry indeed, I was in hurry, my fault. So, we ~~have~~ had :-) some nearly 32 thousands spam accounts. Somebody should say if there are some pages that have to be deleted - the spam category is empty, and if I go through the user log so I see red user pages only (I have just try to see some examples). So the next step should be the captcha and then - no risk no fun - we could try to activate the registration agin... -jkb- 13:10, 14 September 2013 (AKDT)

Good news! So our Main Page's image comes true once more:

.

Lots of thanks to all the contributors to this. (the 32,000 included?) --Zephyrus 00:40, 15 September 2013 (AKDT)

Look Homeward, Angel

This is the novel by Thomas Wolfe, and there is no doubt that it belongs here. What raised my attention is that the djvu file uses an extraordinarily huge amount of bandwidth, 26.89mb per page. The proofread pages are black, but the OCR side looks ok. It was downloaded from Delhi University Library where it was scanned at 6000 dpi! For a novel 300 dpi should be enough. What's th best way of dealing with this? Eclecticology 17:06, 9 September 2013 (AKDT)

I wanted to download it to my PC and to decreas the pixeling, but I didn't succeed. Another possibility would be to download it from the source once again. -jkb- 00:40, 10 September 2013 (AKDT)

This djvu file is being downloaded multiple times, mostly from Chinese IPs. This accounts for 68% of our bandwidth usage. No other djvu file has such results. I don't know what they are trying to accomplish with all these downloads, but I think there is some security risk involved. The pages have not been proofread, and are unlikely to be proofread soon. We could use the "subst" function to move the content to the relevant wiki text pages, which in turn could be marked as unproofread. These djvu files could then be deleted, closing the security hole. Eclecticology 12:33, 14 September 2013 (AKDT)

next step - ?

Hi. Are there some developments in the case of new captcha? For this small domain I would even wuggest to install the email one. As far as I understood it, we need a wikilivres email address, where a new user sends us a mail, and we set his/her accont free. I think we do not have so many serious registrations, so that we would need some say three or four trustied people who have time and visit this domain often - who would have a look for new mails. Or what are the other proposals? Regards -jkb- 08:28, 26 September 2013 (AKDT)

Yes, what about e-mail - it's a good idea. Electron ツ ➧☎ 06:01, 27 September 2013 (AKDT)

I have access to the e-mail account and can register new user accouts. So, I think that it is time to start... Electron ツ ➧☎ 14:01, 21 November 2013 (AKST)

I still have some difficulties to entry - I will email you tomorrow. -jkb- 14:18, 21 November 2013 (AKST)

Try to instal Mozilla Thunderbird and it finds the e-mail account parameters automaticly. It is advice of Eclecticology and it works in my case. Electron ツ ➧☎ 07:24, 24 November 2013 (AKST)

Search phrase

Can anyone explain why the Russian phrase "заплаканная осень как вдова в одеждах черных", from an Anna Akhmatova poem, is used so much as a search term? So far this month, it has been used 751 times, representing 20% of all searches by phrase. The second most used phrase, "воронежские тетради мандельштам" has been used only 38 times. Eclecticology 00:34, 11 October 2013 (AKDT)

Google Translate says "Fall in tears as a widow in black robes". Yann 01:52, 3 November 2013 (AKDT)

Well, nothing extraordinal -- this phrase seems to be just an ordinal beginning of just one of the ordinal poems written by Anna Akhmatova... However, from the IT and Computer Security points of view, one can imaging the following cases for this anomaly:

You should analyze the distribution of the requests for this phrase among IP addresses, subnetworks, probably autonomous systems and periods of daytime.
So, if all of them go from a single computer -- this is probably just a tab with this request, saved in a reader browser, who frequently reboots the computer so all the time he opens the browser you obtain this request...
However, 751 reboots per month seems to be too high... May be it's a buggy browser at the user side crashing frequently (as I have had at Ubuntu after upgrade to 13.10, hehe -- crashed on almost every page with flash video, so I restarted it multiple times a day, opening all preveously opened tabs again and again), or -- this may be a search phrase entered once at a computer at a school class, then this phares was repeated by some of the nearest students -- so if schoolar computers are shutdown at breaks and up again at every new lesson, reading again and again this search phrase at a saved browser tab... In this case, you should see the appropriate distribution among time and the requests should go from same external IP address...
Do you see any differences in letter sencetivity in this search phrase, or different number of commas or blanks? If so, there may be a group of students somewhere deep in Siberia who are forced by their teacher to anaylze this poem or memorize it... So they shared this searchphrase through icq or social network... In this case the requests should go from the geographicaly close subnetworks...
The latter case I see -- if you experience a DDoS- or spam-attack during this period of time, this search phrase may be a part of DDoS attempts (as an implementation of attack on wiki-backend, database and applications) or used by the attacking botnet to verify if your site is still alive... In this case you will see the wide distribution of requests among time and locations...
By the way, after you published this question here, you may see so-called habr-effect (named on a popular Russian IT blog portal and social network habrhabr) -- there, if someone is asking about problems or attacks on a site, the site then obtains even much more serios DDoS since every reader wants to see what is happening with this site and what this site is... ;-)) So you could probably see similar activity after you published this question (in much less scale since this page seems to be not quite active...). Cheers, Hinote 16:02, 7 December 2013 (AKST)

Meta data placement content and format

This issue has come up in a discussion that I have been having with User:Electron. How do we treat meta data for a content page? Where do we put it? What do we include? How do we format it? How do we balance a common approach here for all languages with something that is reasonably compatible with the Wikisource for the language of the material being presented.

Placement: There are three possibilities here: at the top of the content page, at the bottom of the content page, and on the talk page. I very much prefer having most of the meta data at the bottom of a content page. The top of the page should include only a minimum of essential data: title, author, translator, relation to the rest of the work, and a link to the rest of the data at the bottom of the page. For sub-pages I like the concept now used on Czech pages to lead to the metadata on the higher ranking page. Sometimes other information may go there, but I would view that on a case by case basis. The talk page should only need to be used when detailed explanations or histories are required.

Inclusions: For now this can remain open ended but should include when the work was first published, what edition was used for our version, the source for our version (which could be a hard copy), a copyright notice, the proofread status along with who were the two proofreaders. Eclecticology 20:27, 13 October 2013 (AKDT)

I prefer the bottom of the page. At the top is intrusive, and people don't read talk pages.--Poetlister 08:22, 18 October 2013 (AKDT)

Merry ...

File:500px-Xmas tree animated.gif

Merry Xmas
to all of you from
-jkb-

Thanks! :) --Zephyrus 10:07, 24 December 2013 (AKST)

Thank you and I wish the same to you :) Electron ツ ➧☎ 10:17, 24 December 2013 (AKST)

Thank you all. I look forward to continue working with you in the new year. Eclecticology 23:18, 24 December 2013 (AKST)

Happy New Year to all! Yann 01:01, 2 January 2014 (AKST)

Thanks and I wish the same to you :) Electron ツ ➧☎ 05:43, 2 January 2014 (AKST)

Yepp, a nice new year without 25000 accounts of a spamming bot!!! :-) -jkb- 05:47, 2 January 2014 (AKST)

The best new year for all the keepers of Wikilivres , (and for the spambots too, but in some other place, some special place created to make the spambots happy...) ;) --Zephyrus 13:44, 2 January 2014 (AKST)