The Reason No Drupal from me Recently...

Sabina Rose

Sabina Rose
May 22, 2010
2:28 AM
8 lbs, 2 oz

How to Handle Large File Uploads?

I'm actually posting this as a question. If you're looking for the answer, sorry I don't have it yet.

How can we reasonably handle large file uploads? I'm talking in the >100MB range; YouTube, for instance, now supports 2GB files, and this will become increasingly the norm. I don't think that most servers are up to that yet, particularly if you need an application to scale.

Elephant on a Bike

Currently, using PHP, you need to set memory_limit to more than twice the upload_max_filesize, which as you can see would be prohibitive in the example of 2GB uploads; you'd need to set your PHP memory to >4GB (adding the buffer of 64M or whatever you need to run Drupal). EDIT: Looks like I was incorrect in my assumption; if you're not going to process the file, you don't need a huge memory footprint just to handle the raw uploads. Thanks Nate and Jamie!

Even if you manage to have that kind of resource available, you can probably expect things to splode with concurrent uploads...

So I spent some time yesterday looking at SWFUpload yesterday (module here), as I'd misunderstood its claims. Yes, it handles large file uploads (from the browser's standpoint), but you still need to set PHP memory accordingly. Not suitable for what I'm looking for, but it is a really nice way to handle multiple uploads. WARNING: I also learned from experience and much head-scratching that it doesn't work if you have Apache authentication on your server...

Now I'm looking at node.js as a possibility. This looks really great, and might do the job. Basically, it's a JavaScript application that sits on your server. Yes, you heard that right. Turns out that as JS has evolved, it's turned into a really tight language, and should be quite suitable for concurrent tasks.

Sorry if you came to this post looking for answers; I've simply postulated more questions. But I'm hoping that someone with more experience with this issue might be able to comment, and we'll all benefit from it. Additionally, this might turn out to be a handy addition to the Media suite, perhaps as a fancy stream wrapper for handling large files? And I'll definitely follow-up when I figure out how best to tackle this.


Media DX Summit 2010

MDX 2010 FTW!

With a Drupal Beta planned for May 21, the time is coming for a Media Beta as well! Before we can do that, however, there are a few loose ends to tie up. I've identified two critical pieces for a happy Developer's eXperience (DX) before I'll be happy doing that. There are certainly more: see the Media issue queue for more.

MetaData Handling

The Media module creates Media entity objects, which are fieldable. That means we can already attach any fields or taxonomy to any media object, which goes a long way towards handling metadata. However, Media metadata needs are variable and complex. For instance, a field might be fine for adding a taxonomy vocabulary for Video genre or Bird species, but you would need something better if you want to automatically add video duration, YouTube categories, or grab a music file's getID3 data. Basically, we need a larger discussion of what's necessary, what's possible, and how we get there. See this Media metadata issue for more background.

Display Formatting

Currently, we're using the Styles module to power display formatting. We may or may not continue using that. In any case, we'll need to ensure the formatters more closely follow Media Types, and we'll need to offer a pluggable UI for changing formatter style presets, similar to Image Styles (Imagecache in core, for those not yet familiar).

Media DX Summit 2010?

I'd love to lock up some fellow developers for a couple of days in a room to bang on these ideas. At the same time, my partner Gwen is due on May 22 for our second child, so firstly, I can't really travel any time in the foreseeable future, and secondly, even if we had a summit here in Harrisburg, it would either have to be like this week, or in mid-summer. Considering the deadlines involved for this, we need to get cranking. Thus, the summit I would love to see happen will probably either have to happen remotely, or perhaps without my involvement. :(

Anything Else?

Are there any other issues you would suggest to be critical beta-blockers? Do you have any thoughts to add to the issues I've suggested? Please add to this thread!

Aaron Winborn

(Cross-posted at g.d.o.)

XMPP and Drupal developers' round table

It's at if you're interested in learning more about the future of XMPP integration with Drupal. The event will be on Friday, March 26, at 12PM EST (17:00 CET).

Kristof wrote:

Recently a lot of people started are working concurrently on XMPP and Drupal integration. So I thought it would be a good idea to share our ideas so we can work on top of a common platform.

We are going to have the meeting on Dimdim. You can join the event at

Avoiding the One Month No-Blog-Post Anniversary Blog Post

Inspired by the State of Rob Loach Address, I decided I don't particularly want to let a year go by without blogging, especially considering I don't have the excuse that I'm micro-blogging. So without further ado, here's what I've been up to this month: cat nap
  • Media module, of course. Though I'm not in the swing I was in January. I've worked some on an experimental browser that allows more FAPI magic than the current one that's a Javascript framework (which is thus largely inaccessible to many Drupal developers, though comes with its own advantages). Other than that, it's just been some code clean up.
  • Embedded Media Field now supports RTMP streaming servers. Also, Media: YouTube has been ripped from the dev-2 version of it, and now supports iPhone. (I can't really take credit for either of these new features, other than for committing the patches, but it makes it look like I've actually been working...)
  • I started a new blog. Well, it's in its conceptual and wireframe stages still, so no announcement yet... Has to do with raising children to be entrepreneurs. We'll see...
  • Been working on some game stuff, with the new Game Utilities: Quest module, which uses a sound PHP object framework to expose quests to characters.
  • XMPP! This is awesome! Through work at Advomatic, I've set up an XMPP server and am working on a fancy Drupalized/themeable client. Post soon!
OK, that's all for now. Time to wake up Ashlin!

Drupalcon: Free as in Beer!

Druplicon on Trolley

Yes, you heard it right! I'm not going to Drupalcon after all -- it's too close to the due date of our next baby. As much as I'd like to go (Drupalcons really are all that), I would be mortified to miss the birth of a daughter, if she decided to come in to the world a little early.

I'm going to give my ticket to a worthy participant in exchange for a couple of hours of review of the Media module. (Don't worry if you're not a developer or know how to patch or anything -- many of the issues are around usability, so we appreciate all levels of review.) Just contact me off-list if you're interested. You'll still be responsible for travel, though you might be able to find sponsorship for that as well if you act quickly.

That means that, obviously, I'm not going to be able to come present the awesome work from the team developing that module in SF. Don't worry -- there is a panel in the works with some of the biggest names in Drupal multimedia development. I'll make sure to let you know here when the details of that have been settled!

Drupal Media + YouTube + WYSIWYG

Thanks to some fine work by Jacob Singh, Dipen Chaudhary and myself over the past month, we've gotten Media working fairly well with WYSIWYG, with Media: YouTube and Media: Flickr to boot.

WYSIWYG Browser Thumbnails

Click through to see the action...

Drupal Media, Now with Fieldable Entities!

If you haven't looked recently, there's been some huge progress recently for Drupal's Media module. Jacob Singh from Acquia has jumped on board, paving the way for fieldable entities! This allows Media asset objects to be a first class Drupal citizen, alongside Nodes, Users, Taxonomy, and Comments. (Hopefully in core for Drupal 8!) Also, Dipen Chaudhary has been hard at work providing WYSIWYG support!

Media as an Fieldable Entity

Read more for a tutorial...

Media Styles

I'm adding display formatters to the Media module, and could use some feedback.

Basically, I'm taking the work from the Image Styles built into core (which is a port of Imagecache), and building a wrapper around it. A Media Style would be a collection of styles, based on the stream (public://, private://, youtube://, etc) and file mimetype (image/jpeg, image/png, etc.), that would be applied to a specific filefield display (either in the node teaser/page display, or a view field display, or possibly other places, such as inline).

As an example, you might have a 'small-box' Media style that contains a 'medium' image style, a 'preview' youtube style, and an 'inline' pdf style. Thus, if the filefield contained an image, it would display it with the image scaling, a youtube video with a small player, and a pdf would be displayed in an iframe. Undefined streams/mimetypes would fall back to the default file listing.

The module is intended to work stand-alone, with File, Image, and/or Media. Thus, one question I have is if it should be bundled with the Media module, or packaged outside the module. On the one hand, as it can be run w/o Media, it might be useful in other situations. On the other hand, I imagine 98% of Media users would also want this module, so I'm hesitant to create a new external dependency. I've mostly decided to bundle it with the Media module, but am open to new considerations I haven't had yet.

I've included two screen shots. The first screen shot at the top shows current functionality. Clicking on a radio will automatically load a new preview of the selected style (that part's not built yet, but that's the idea).

The second below was the original display for administration. It's now actually using vertical tabs, as with the first, but I'm including it since it gives a large overview (and is what it would look like w/o javascript anyway).

I'm mostly looking for feedback of how the administrative UI could be improved, as well as how to word instructions to the user that won't scare them off. Also, for any other conceptual, architectural, or other comments.


(Cross-posted in the Drupal Media Group.)

Continuing the Good Fight against Spam...

So I installed both Spamicide and Hashcash, and Mollom reported a huge dip in spam attempts in August (from the 17th to the 28th, as seen in the screenshot). However, the amount of spam getting through steadily increased after that. The volume of visits according to Google Analytics has remained about the same over the past year. Now the volume of spam getting through is back to roughly where it was before I made the configuration changes...

Mollom Spam in August

I'm about to give up and look at Akismet or Typepad (using AntiSpam for Drupal users)...

I'll keep folks posted.

Syndicate content

The Society for Venturism has chosen me as the recipient of its charity for this year, to hopefully offer me cryonic preservation when the time comes. And this month, Longecity, an excellent forum for the discussion of issues related to extending the lifespan of humans, has offered up a matching grant of up to a thousand dollars to help out! So help out! Please.