Tech

PayPal in Vietnam

Submitted by tomo on October 5, 2012 - 4:11pm

Search for PayPal in Vietnam and you won't get many good results.

If you're American or from another Western country and just traveling to or working in Vietnam temporarily and you already have a PayPal account then heed this one warning: DO NOT access your PayPal account WITHOUT a VPN! (If you don't already have a VPN get one. Ref: StrongVPN) Doing so will flag your account (similar to what happens when you use your credit card overseas without warning your bank first) and your PayPal account will be limited to receiving money only until you "verify" it. This can be a PITA so just DON'T access PayPal using a Vietnamese IP address. You can Google "what's my IP" to make sure your traffic is being tunneled properly.

History: PayPal started as a service for eBay sellers who didn't have real businesses and therefore couldn't open merchant accounts to accept credit cards to accept payment from buyers, both as bank transfers as well as credit and debit cards. It was eventually bought by eBay due to the symbiotic nature of the two. For many years PayPal was not available to Vietnamese users. due to the high amount of fraud originating from Vietnam. Fraud is one of PayPal's biggest problems and they spend a considerable amount of resources trying to detect fraud, often resulting in suspended accounts of legitimate users. Many non-Americans can't use it which has resulted in services like Moneybookers or Liberty Reserve.

PayPal's absence from Vietnam left open an opportunity for Vietnamese PayPal clones like MobiVi, etc. The problem with these is that none are as versatile as PayPal and some only allow transferring money between users of the service who also use the same bank!

If you're Vietnamese or working in Vietnam and want to open a Vietnamese PayPal account it is now possible to do so and receive money. You simply need to open a debit/credit card account at one of these banks: Techcombank, ACB, Đông Á, Eximbank. Then you can connect it to a Vietnamese PayPal account. Remember that PayPal won't allow you to change your country once you open an account.

Final warning: Don't access your US PayPal account from a Vietnamese IP address. Always use a VPN like StrongVPN. I learned the hard way and ended up with a locked and useless PayPal account. Whether you should even use PayPal is another issue altogether.

Archiving Twitter to Blog Post

Submitted by tomo on October 5, 2012 - 10:30am

Step 1: Install t (Ruby Twitter command line client). Run the following (substitute your own Twitter ID):

t timeline @tomosaigon --csv --number 3000 > tweets.csv

Step 2: Convert CSV to JSON using http://www.cparker15.com/wp-content/uploads/csv-to-json/csv-to-json.html

Step pre-3:

cached=$(/usr/bin/grep $1 $CACHE)
if [[ $cached != "" ]]; then
        echo $cached|/usr/bin/cut -d' ' -f2 ; 
        exit
fi
 
url=$(/usr/local/bin/curl -v $1 2>&1|/usr/bin/grep '^< Location: '|/usr/bin/sed -e 's/< Location: //')
echo $1 $url >> $CACHE
echo $url

Step 3: Open Chrome Developer Tools and in the command line type "var a = " then paste the copied JSON. Hit enter.

var out = '';
for (var i = 0; i < a.length; i++) {
    var t = a[i]; 
    var d = new Date(t['Posted at']); 
    if (d >= new Date('2012-09-01 00:00') && d <= new Date('2012-10-01 00:00')) {
        var tweet = t['Text'];
        var tcos = tweet.match(/http:\/\/t.co\/\w+/g);
        if (tcos) {
            for (var m = 0; m < tcos.length; m++) {
                tco = tcos[m].replace(/\//g, '%252F');
                $.ajax({url:'http://www.saigonist.com/saigonist/untco/' + tco, async:false, success:function(data){
                    tweet = tweet.replace(tcos[m], data.trim());
                }});
            }
        }
 
        tweet = tweet.replace(/("")/g, '"').replace(/(^|\s)@(\w+)/g, '\$1<a href="https://twitter.com/\$2" rel="nofollow">@\$2</a>').replace(/(^|\s)#(\w+)/g, '\$1<a href="https://twitter.com/#!/search/%23\$2" rel="nofollow">\$2</a>')
        out += ((d.getMonth() + 1) + '/' + d.getDate() + ' ' + d.getHours() + ':' + d.getMinutes() + ' ' + tweet + ' ' + '<a href="http://twitter.com/' + t['Screen name'] + '/statuses/' + t['ID'] + '" rel="nofollow">#</a>');
        out += '\n';
    }
}

Bypass Basic Auth for Localhost on Nginx

Submitted by tomo on October 1, 2012 - 12:45am

During development sometimes I hide a server behind a shared password for http basic auth, based on an Apache configuration (if you still want to use .htpasswd but don't have Apache and htpasswd installed you can still generate the password entries using openssl). nginx can use the same format of .htpasswd files by using the HttpAuthBasicModule. But sometimes I need a service to crawl pages on the server (for example Drupal's linkchecker module). And sometimes storing the basic auth login and password in a script isn't wanted or advisable. Instead we can configure nginx to skip authentication when connecting from localhost/127.0.0.1.

Open up your nginx configuration which already contains your auth_basic directives. Assuming you are running a PHP site and you have a "location ~ \.php$" block, make it look more like this:

location ~ \.php$ {
  satisfy any;
  allow 127.0.0.1;
  deny all;
 
  auth_basic            "Restricted";
  auth_basic_user_file  /yoursite/path/.htpasswd;
}

Remember that the order of the allow/deny directives matters. More on the nginx_http_access_module module.

How can we store tables or spreadsheets in Drupal content nodes?

Drupal is a Content Management System as well as a Content Management Framework. It's meant for facilitating the creation and editing of content, where content doesn't just mean simple text. CCK allows Drupal site managers to easily enable complex data types for their content. But sometimes it's not so easy to manage lots of simple data in the way that a spreadsheet (meaning Excel or Google Spreadsheet) easily manages many rows and columns of related data.

Sometimes you really want tabular data and you may not know for sure how many rows or columns you'll want beforehand, which is never a problem for Excel or Google Spreadsheet (which is also why sometimes people abuse Excel as a general purpose database). Maybe you are importing spreadsheets or are using spreadsheets with many rows and columns, and only want to use a section of the spreadsheet. Maybe you need to use spreadsheets because you are using formulas and want to do some graphing based on some numbers too. In fact, Excel can do much more. Rather than hoping all of those use cases will make it into a Drupal module anytime soon, wouldn't it be nice if you could just embed a spreadsheet in a node?

There are some modules for attaching tables or tabular data to nodes but in the end the interface can be a bit unwieldy. I discussed building something like a spreadsheet to quickly edit many values and below I present a demonstration.

What is already available that we could use to attach spreadsheets or something similar to nodes?

1. You could: Configure a Google spreadsheet that's either public or somehow accessed via a Google api, perhaps using OAuth2 or just assuming the viewer also has edit access to the spreadsheet. Google already lets you embed spreadsheets including the editable spreadsheet but they don't let you limit the view of the spreadsheet (limiting the view still lets everyone access the whole spreadsheet by changing the URL).

We would also need new nodes to automatically create sheets in the spreadsheet or new spreadsheets altogether and embed them in node edit. This would need to use their APIs since the embeddable Google Spreadsheets require an existing spreadsheet, otherwise this step needs to be done manually for each node - a new Google spreadsheet created per node.

This Redmine Google Docs plugin takes a similar approach and shows how the resulting data could be embedded.

(This might be worth investigating as well.)

2. Another approach would be to take an existing JavaScript or Java spreadsheet that could be embedded. This could be exactly what you want if you need the full functionality of Excel in each node. One such online spreadsheet is ZK Spreadsheet which is written in Java, and would let you have full formula and charting support.

3. Embed a widget like the form below as a CCK field in a node. It would be a new CCK field type where the editing widget is the below spreadsheet-like table and the display widget might be the same thing, an uneditable table, or the raw JSON string representation. In the database, the actual value of the field would be JSON (unless you wanted to use PHP's serialize() which would require POSTing values first instead of just posting the json value). This means you wouldn't relate or compare this field to anything else, as with any serialized data in a database. You could come up with some other schema involving columns and rows to store the table's values for each field in a node but I'm not sure it's worthwhile.

I created this demonstration but haven't turned it into a Drupal module. There is code to dump the edited values into a single JSON value to be stored in the database but the storage would be module-specific.

Get the JavaScript - then combine it with html and CSS, which you also see below.

.
A
B
C
+
1
2
3
+
JSON: "

"

Drupal ImageCache Image Quality

Submitted by tomo on September 26, 2012 - 11:04am

The problem is this. ImageCache is a great simple way to transform random images into useful dimensions, etc. But sometimes the uploaded image is already exactly the quality and even dimensions we want. But after imagecache module runs, the file size is actually larger than the original while the quality has gone down, despite JPG image quality being 100% (default is set to 75%). One workaround can be to re-sharpen the image (imagecache_sharpen) but this also loses some quality.

1. Check that the uploaded image (metadata saved as CCK field, file stored directly in sites/default/files) has not been degraded in quality.

2. Check that minimum/maximum resolution for the image fields (CCK) are set to 0 (no restriction) so that they aren't accidentally resized at the upload to CCK step.

3. Run "drush imagecache-flush" in case you have files left over from a recent imagecache configuration change.

4. Try replacing GD with ImageMagick. If you stick with GD, and try compiling a newer GD library. ImageMagick by default can output higher quality images than GD.

How does Drupal's image resizing and processing work?

ImageCache goes through imageapi which uses different image processing functions based on the library (by default GD and ImageMagick support are provided) used.

Any resizing operation will result in a change (loss) in quality. That is to be expected since upsizing creates new pixels from single original pixels without having new information (unlike the magical effects of zooming into images on CSI shows) and downsizing will generally cause information to be lost and many pixels will "store" the information from several surrounding pixels.

So only operations that keep the image the same dimensions or crop the image (in which case the cropped portion has the same dimensions in both old and new files) will retain the same quality and avoid any blurring.

When using PHP's GD, Drupal will make calls to imagecopyresampled which will affect image quality unless image width and height are kept the same or cropping is performed. This is partly due to imagecopyresampled and partly the way it is called. "Fastimagecopyresampled" is a possible replacement: (from http://us.php.net/manual/en/function.imagecopyresampled.php#77679) - this is a higher quality version of PHP's imagecopyresampled. It does result in higher quality but requires hacking Drupal code instead of overriding any hooks.

I've found that it's better to use ImageMagick than GD. The only remaining problem is that for similar quality images, the file size is something like twice as big after ImageMagick processing. You could mess around with ImageMagick's quality settings by hacking Image API code or by copying imageapi_imagemagick.module into a custom module and configuring it for use instead. But you risk breaking quality for other images. I haven't found the reason why IMagick makes image file sizes larger.

Telecommunications in Vietnam visualized with some network maps showing both international and domestic peering arrangements between ISPs. Bookmark this page and I'll add more maps as I find them.

For international bandwidth VNPT (VDC) is important. For domestic internet look for connections through VNIX. VNPT, FPT, Viettel, NetNam, SPT, even VinaGame and Google.

Information about server hosting in Vietnam
Information about cloud hosting and content delivery network access from Vietnam

I've been doing some testing on a plethora of content delivery networks using cloudharmony.com's speedtest service. Testing from multiple ISPs in Vietnam, during the most recent undersea cable outage, it becomes clear that, although none of the CDNs publicly have datacenters in Vietnam, a small handfull of networks like Edgecast CDN (via http://www.speedyrails.com/) come out on top as winners (except for an anomalous reading of small file size downloading from one ISP). In these tests, performance of poor hosts relative to the top was as much as 20 times slower, meaning CDN selection without taking into consideration the real world results in Vietnam can mean you're throwing a lot of money away. Poorly performing content delivery networks were more or less not any faster than hosting in the US.

Limelight CDN also had some of the best overall results, better than Edgecast in some cases.

VoxCAST CDN had a surprisingly low latency time suggesting it could be closer to Vietnam than anything in Singapore, but VoxCAST (Internap) do have a Singaporean datacenter which is probably just faster than other points in Singapore. Unfortunately, the transfer speeds were too low. Internap's CDN taken into consideration alone fared poorly. Edgecast state they use Equinix Singapore Exchange so it may be worth looking into colocating there.

Windows Azure CDN had some promising results in some tests whereas other tests failed. I didn't investigate further when CDNs failed, assuming they should not fail.

What may be more surprising is who isn't the winner here: Amazon

Amazon EC2 instances in both Singapore and Tokyo (just for comparison) performed poorly both in terms of latency and throughput. Amazon's CloudFront service did no better.

Asia-America Gateway undersea cable outage

At the time of writing, the AAG internet cable that connects to Vietnam at Vung Tau's landing station, thus providing FPT (although Viettel and VNPT do also use AAG) in Ho Chi Minh City and its subscribers with international bandwidth to Hong Kong, Singapore, and points beyond in the US, has been cut or damaged again. The cable experiences cuts both at the Vung Tau cable landing station as well as in the main trunk when all traffic along the cable would be affected. It is one of several international links to Vietnamese ISPs such as SEA-ME-WE 3 (but not SEA-ME-WE 4) which connects to Vietnam in Da Nang and links Southeast Asia with East Asia, Australia, the Middle East, and Western Europe. TVH and TGN-Intra Asia Cable System also land in Vung Tau connecting Southern Vietnam to the greater Internet. Vietnam also has terrestrial internet connections to China.

[Update: The AAG cable is being repaired from October 25, 2012 until November 4.]

About Saigonist Mutual Friends Facebook App

Submitted by tomo on August 20, 2012 - 9:22am

My friend Grover and I have a lot of mutual friends as I noticed on Facebook one day. And I know that I have a lot of friends with whom I share a large number of friends. I got curious and asked myself: "Who do I have the most shared friends with?" This question turned into a Facebook FQL query which gave me the data. Then I decided to turn it into a "product" and ended up learning stuff about Facebook's PHP and JavaScript SDKs.

The final result is this Facebook App: Saigonist Mutual Friends.

Just enable the app and you'll see a list of your top 10 friends with mutual friends. I got my answer: my friend Jodie and I share 158 friends.

--

Next steps? I'd like to visualize what cliques or subnetworks exist within my Facebook network...

Writing Vietnamese is a pain. You have several different input systems to choose from (VNI, Telex, VIQR, etc.) and they all require some effort to memorize and get used to. But as it turns out most Vietnamese text can be understood without those accent (diacritic) marks at all. Vietnamese people are accustomed to texting (sms) each other using Vietnamese words written without marks - because their phones often don't support Vietnamese characters. But it's better to have the marks and emails and posts on forums often do. GMail even supports a Vietnamese software keyboard built into the interface now.

But if Vietnamese people can understand Vietnamese without diacritics, can computers? Turns out there is software that can take unaccented Vietnamese text and ADD the diacritics!

Let's take some text:

Chuyến phiêu lưu khám phá bỏ nhà ra đi đầu tiên của bạn vào năm bao nhiêu tuổi? 
- Nghĩ lại thì thấy hồi xưa mỗi lần mà bị mẹ la là hay giận, bỏ nhà đi lắm, vì lúc nào cũng nghĩ mình đúng hết. Giận thì giận nhưng mà đi lang thang rồi lại về, hoặc về trong tình trạng được tìm thấy và lại tiếp tục bị mắng :D

Then we strip the accents and put it into a few websites to see the results.

http://vietnameseaccent.com/

Chuyến phiêu lưu khám phá bỏ nhà ra đi đầu tiên của bạn vào năm bao nhiêu tuổi?
- Nghĩ lại thì thấy hồi xưa mỗi lần mà bị mẹ là lạ hay gián, bỏ nhà đi làm, vì lúc nào cũng nghĩ mình dùng hết. Gian thi giản nhưng mà đi lang thang rồi lại về, hoặc vê tròn

http://vietlabs.com/vietizer.html

chuyến phiêu lưu khám phá bỏ nhà ra đi đầu tiên của bạn vào năm bao nhiêu tuổi?
- nghĩ lại thì thấy hồi xưa mỗi lần mà bị mẹ là là hay gian, bộ nhà đi làm, vì lúc nào cũng nghĩ mình đứng hết.
gian thì giản nhưng mà đi lang thang rồi lại về, hoặc về trọn

http://www.easyvn.com/tiengviet/index.php

Chuyến phiêu lưu khám phá bỏ nhà ra đi đầu tiên của bạn vào năm bao nhiêu tuổi?
- Nghĩ lại thì thấy hồi xưa mỗi lần mà bị mẹ là là hay giận, bỏ nhà đi lắm, vì lúc nào cũng nghĩ mình đúng het. Gian thì giận nhưng mà đi lang thang rồi lại về, hoặc về tron

The results are nearly the same except for the last word: 'tron'. This is because the real word is "trong" but it got cut off in the de-accenting process! So each different software took a different guess as to what the word was, but it was the wrong word to begin with.

All in all, they do a pretty good job and probably better than even some native Vietnamese speakers due to the fact that some tones are mixed up!

Today I spent the day hacking on the WordPress site for BarCampSaigon. I'm no expert on WordPress but trying to hack a theme with it makes me appreciate Drupal so much more. WordPress does have an "API" or at least some documented functions which are used internally which can also be used by developers but it's certainly not designed with developers in mind. Wordpress is great for bloggers. Drupal is great for developers and users who need something more than a blog will have to choose between working with Drupal or fighting against WordPress. :-)

Due to two recent conferences, BarCamp Hanoi and the opening of DrupalVietnam.org, I put together a presentation called "The Business Case for Drupal in Vietnam". I'll blog about those two events later.

I targeted two main groups: outsourcing companies based in Vietnam who want to attract more clients abroad, and any software development shop in Vietnam who is deciding what technology to use to develop websites. There is a third group, who are those companies with simpler website needs who perhaps only need one website and are not in the business of making websites.

I go over a number of common concerns that customers outside of Vietnam might have which Vietnamese companies might not expect. One point is being vendor agnostic when developing what is essentially a CMS. I make a strong point that one should never fall to the temptation of developing your own in-house CMS and as a consumer, you should stay far away from such "bespoke" solutions in the modern age where content management frameworks such as Drupal exist. I also think Drupal represents a strong brand name that is not well-known yet in Vietnam, but we all know how much Vietnamese people love brands.

On the supply side, I talk about why Drupal is a decent choice for Vietnamese developers, similar to any web developer. But the main recent news is that there is finally Drupal training being made available in Vietnam, much like for Joomla, and that PHP is widely known in Vietnam not just because there are books on it in the Vietnamese language, and that there is now a core Drupal community in Vietnam represented by DrupalVietnam.org (which I somehow became vice president of).

In general, I say go with your strengths. If web development is not your strength, then hire someone proper to do it for you while you focus on your core business. If web development is your business, make better use of your developers by using a CMS.

Anyways, here's the presentation.

Syndicate content
© 2010-2014 Saigonist.