Getting Started with RETS for PHP
Update: Docs
Since the time of writing this article, the places in which I link to very pertinent, very helpful RETS references are now gone, slop-handedly replaced with a ridiculous PDF. That’s right—whereas before you had an entire website to search through and reference and link to, now all you have is a bloody downloadable document to Ctrl+F your way into madness. So I apologize in advance for any links to DMQL, for instance, which drop you off in the middle of the Slough of Despond.
Any suggestions to better documentation are welcome, and any complaints about RESO’s documentation should be sent to [email protected] (yes, I have already tried to contact them about the old documentation, but to no avail).
Getting Started
Note: R-E-A-D your way through all four exercises. I’ve tried my best to keep it short, but there is an order to follow which will lead you into understanding your own MLS server and how to start your own app.
There aren’t many good articles out there on working with RETS. The short of it is: you will have to learn a lot for yourself. This guide is intended for moderate-level programmers that just need to get to a place where they’re comfortable handling data. In this tutorial, you’ll be tackling:
- Seeing the tables on your server
- Seeing the table columns on your server
- Querying the RETS server for listings, based on data from Step 1 and Step 2
- Downloading photos for a listing
To start, a couple of terms:
IDX Internet Data Exchange. Just think API. No, it’s not an API format; it’s simply a term that tells you data is being transferred, like an API. That’s not helpful. I know. This just gives realtors a handle to try and understand what you, the programmer, are doing. Just think of it as a realtor’s way of saying API.
RETS Real Estate Transaction Standard. Just think SOAP. Again, it’s not SOAP, but it’s an XML-encapsulated response that you’ll be working with to get your data. Wait, I’m not working with JSON!? Don’t get me started.
Setup
For this example, we’ll be using PHRETS. It’s a lightweight, one-file RETS library, has decent documentation, and pretty much works right out of the box. Plus, it’s still actively maintained on Github. You can download it with my example files.
The next step is getting a login to RETS. This is handled by your local MLS. In Orlando, you’d be dealing with MFRMLS. Part of the process is developing a relationship with your MLS IDX team, and being prepared to call them a number of times to get what you need. Be patient, and understanding, and they’ll go out of their way to help you. I also assume you’re working with a realtor/broker to get this, and you’ll need to work with them too to fill out their necessary paperwork to get access. Through a series of phone calls and costs, you will eventually wind up with three things:
- Login URL (this is different for every MLS)
- Username
- Password
Example 1: Connecting and Viewing Server Contents with GetMetadataTypes()
Download my example files here if you haven’t already. You’ll find the PHRETS files in /lib, as well as some starter code. Open login.php.
/////////////////////////// LOGIN INFO ///////////////////////////// $login = 'http://yourmls.com/Login.asmx/Login'; $un = 'RETS000'; $pw = '##########'; ////////////////////////////////////////////////////////////////////
Edit this and save with your credentials from the previous step. Note that your URL might look completely different, but it should end in /Login.
Now we’re going to see what’s on the server. One thing to keep in mind: your RETS server might look completely different from somebody else’s. Even with a standard, databases can be set up differently. We’ll be going through a process that helps you find out what you’re working with, so expect your results to be slightly different from what I see on my server. Open example1.php. You’ll only see a few lines:
/* Initialize Object */
$rets = new PHRETS;
/* Connect */
$connect = $rets->Connect($login, $un, $pw);
/* Query Server */
if($connect) {
$types = $rets->GetMetadataTypes();
print_r($types);
$rets->Disconnect();
} else {
$error = $rets->Error();
print_r($error);
}
A couple things are happening here, but not too much. You can see the PHRETS object being initialized, and a connection being made to the server. You can also see that I use print_r() like it’s going out of style. The connection request is just a cURL with your login info being sent in the request. You don’t have to know too much about it because PHRETS handles it. If the connection is successful, it will print an array of all the table values; if failed, it will print an array containing error information. The big array of stuff should be laid out somewhat in this pattern:
- Office
- Office
- Property
- Commercial
- Income
- Residential
- Rental
- Lots and Land
- Cross Property
- User
- User
What is all this? This is your key to accessing the property listings. The top-level items are called Resources. The sub-items are referred to as Classes. This is important. Remember this. You’ll see the words Resource and Class appear the more you work with RETS; this is what it’s talking about.
Each Class has its own database table, with its own unique values. For example, this means Residential properties and Rental properties both have to be queried separately. Obviously, how you combine/store data is up to you, but it must be accessed separately nonetheless. Additionally, Realtor Office information is simply there if you need it, and honestly I have no clue about the User section.
Taking a closer look at the actual output from GetMetadataTypes(), each Class should resemble this:
... [3] => Array ( [ClassName] => 4 [VisibleName] => RES [StandardName] => ResidentialProperty [Description] => Residential [TableVersion] => 33.04.13720 [TableDate] => Wed, 19 Dec 2012 05:35:20 GMT [UpdateVersion] => [UpdateDate] => ) ...
We can ignore most of this, but pay attention to ClassName and Description. You’ll need the class name. A lot. In fact, I would recommend saving the whole output from this script somewhere for later, until you’ve memorized the ClassNames on your MLS server (did I mention you’ll be using this a lot?).
Once you know what these 3 things are:
- Resources
- Classes
- ClassNames
You may proceed; if not, review. To continue, let’s see what information is stored in the Residential table.
Example 2: Viewing the Table Fields with GetMetadataTable()
Open example2.php. You’ll find much of the same code, but now instead of GetMetadataTypes() you’ll find GetMetadataTable($resource, $class):
if($connect) {
/* Get table layout */
$fields = $rets->GetMetadataTable("Property", 4);
/* Take the system name / human name and place in an array */
$table = array();
foreach($fields as $field) {
$table[$field['SystemName']] = $field['LongName'];
}
/* Display output */
print_r($table);
$rets->Disconnect();
...
In the last example, we saw the ClassName for Residential was 4 (in some examples you’ll find the VisibleName instead). You can simply run print_r($fields) instead (I love print_r!) to look at all the juicy bits here, but we’re taking a simpler route and only getting the SystemName and the LongName (the human-readable name). It will still be pretty long:
Array ( [sysid] => sysid [1] => Property Type [3] => Record Delete Flag [4] => Record Delete Date [5] => Last Transaction Code [9] => Photos, Number of [15] => Tax ID [17] => Agent ID [18] => Selling Agent ID [19] => County [32] => Beds [33] => CDOM [35] => Agent Name [46] => Zip Code [47] => Zip Plus 4 [49] => Address [55] => Year Built [59] => Status Change Date [106] => Entry Date [108] => Listing Date [112] => Last Update Date [138] => Office ID # [140] => Sold Office ID [143] => Str. Dir. Pre [165] => Street # [175] => ML# (w/Board ID) [176] => List Price [178] => Status [391] => Sold Price [406] => Sold Date [410] => Sold Agent ID [421] => Street Name [1334] => Additional Public Remarks [1335] => Additional Parcel Y/N [1349] => Property Style [1350] => Building Name/Number [1354] => Bonus Room (Approx.) [1368] => Minimum Lease [1374] => # Times per Year [1375] => Taxes [1381] => Dining Room (Approx.) [1384] => Dinette (Approx.) [1397] => Elementary School [1405] => Family Room (Approx.) [1415] => Living Room (Approx.) [1418] => HOA Fee [1420] => High School [1425] => Middle or Junior School [1426] => Kitchen (Approx.) [1432] => Legal Description [1437] => Lot # [1455] => Model/Make [1457] => Public Remarks [1466] => Master Bedroom (Approx.) [1495] => 2nd Bedroom (Approx.) [1514] => 3rd Bedroom (Approx.) [1518] => 4th Bedroom (Approx.) [1522] => 5th Bedroom (Approx.) [1660] => Special Listing Type [1670] => Subdivision # [1709] => Financing Available [1711] => Realtor Info [1716] => Architectural Style [1717] => Additional Rooms [1718] => Location [1720] => Utilities [1722] => Water Extras [1723] => Fireplace Description [1724] => Master Bath Features [1725] => Interior Layout [1726] => Interior Features [1727] => Kitchen Features [1728] => Appliances Included [1729] => Floor Covering [1731] => Heating and Fuel [1732] => Air Conditioning [1733] => Exterior Construction [1734] => Exterior Features [1735] => Roof [1736] => Garage Features [1737] => Pool Type [1739] => Foundation [1743] => Community Features [2292] => Str. Dir. Post [2294] => Full Baths [2296] => Half Baths [2298] => Price Change Date [2300] => Driving Directions [2302] => City [2304] => State ID [2306] => Street Type [2314] => Legal Subdivision Name [2316] => Complex/Community Name/NCCB [2320] => Zoning [2322] => Lot Dimensions [2326] => Square Foot Source [2328] => Total Acreage [2334] => Listing Type [2338] => Trans Broker Comp [2340] => Buyer Agent Comp [2344] => Non-Rep Comp [2346] => Sq Ft Heated [2350] => Water Name [2352] => Private Pool Y/N [2362] => Lot Size [2368] => Office Name [2386] => Sell Agent Name [2390] => Sell Office Name [2455] => List Agent 2 ID [2497] => Virtual Tour Link [2606] => List Agent 2 Name [2620] => Office Primary Board ID [2622] => Lot Size [SqFt] [2624] => Lot Size [Acres] [2708] => MLS Zip [2759] => LastImgTransDate [2763] => LP/SqFt [2765] => SP/SqFt [2769] => ADOM [2779] => Area (Range) [2781] => Team Name [2789] => Balcony/Porch/Lanai (Approx) [2791] => Building # Floors [2793] => Annual CDD Fee [2795] => CDD Y/N [2797] => Condo Floor # [2801] => Fireplace Y/N [2803] => Floors in Unit [2805] => Garage/Carport [2809] => Homestead Y/N [2811] => Maintenance Includes [2813] => Max Pet Weight [2819] => Property Description [2823] => Special Tax Dist.Y/N (Tampa [2854] => Unit # [2856] => Total Units [2879] => MH Width [2899] => Mo.Maint.$(addition to HOA) [2901] => HOA Payment Schedule [2935] => LSC List Side [2945] => Studio Dimensions [2983] => SW Subdv Community Name [2991] => Selling Agent 2 ID [2992] => Listing Office 2 ID # [2993] => Selling Office 2 ID [2995] => Listing Office 2 Name [2996] => Selling Office 2 Name [3010] => Show Prop Address on Internet [3011] => Water View [3015] => Green Certifications [3020] => Selling Agent 2 Name [3021] => Great Room (Approx.) [3022] => Waterfront Feet [3026] => Subdivision Section Number [3027] => Total Building SF [3036] => Housing for Older Persons [3048] => LSC Sell Side [3062] => Special Sale Provision [3063] => Water Access Y/N [3064] => Water View Y/N [3065] => Water Frontage Y/N [3066] => Water Extras Y/N [3067] => Water Frontage [3068] => Water Access [3074] => HOA/Comm Assn [3075] => Pets Allowed Y/N [3076] => Pet Restrictions [3077] => Study/Den Dimensions [3078] => Country [3080] => # of Pets [3084] => New Construction [3085] => Construction Status [3086] => Projected Completion Date [3146] => Planned Unit Development [3147] => HERS Index [3148] => Flood Zone Code [3149] => Land Lease Fee [3165] => DPR Y/N [3186] => Pool [3187] => Public Remarks New [3189] => Condo Maintenance Fee [3190] => Condo Maint. Fee Schedule )
Look at all this! This is all the data you have for every listing! What’s even cooler: from time-to-time, MLSs will add new fields, giving you even more data to work with (but the IDs will usually stay the same)! You have instant access to something most realtors don’t have. Bask in it.
When you’re done basking in the glory of big data, save this output as well because you’re going to be using SystemName values to query listings in the next example.
Example 3: Searching Listings with SearchQuery()
To recap, searching properties requires:
- Knowing your Server’s ClassNames (Example 1)
- Knowing your Server’s SystemNames (Example 2)
Once you have these 2 things, you can ask your RETS server to serve listings. Open example3.php. This time, you’ll see:
if($connect) {
$sixmonths = date('Y-m-d\TH:i:s', time()-15778800); // get listings updated within last 6 months
/* Search RETS server */
$search = $rets->SearchQuery(
'Property', // Resource
4, // Class
'((112='.$sixmonths.'+),(178=ACT))', // DMQL
array(
'Format' => 'COMPACT-DECODED',
'Select' => 'sysid,49,112,175,9,2302,2304',
'Count' => 1,
'Limit' => 20
)
);
/* If search returned results */
if($rets->TotalRecordsFound() > 0) {
while($data = $rets->FetchRow($search)) {
print_r($data);
}
} else {
echo '0 Records Found';
}
A lot going on here. To start, let’s look at SearchQuery($resource, $class, $query[, $options]). You’ll notice 3 main parameters, and an options parameter. The first 2 you know already. The third parameter is a Query in DMQL (Docs Here). A typical DMQL Query looks like:
( (FIELD_ONE = VALUE), (FIELD_TWO = VALUE), (FIELD_THREE = VALUE), ...)
That would be the MySQL equivalent of:
select * from [whatever] where FIELD_ONE = VALUE and FIELD_TWO = VALUE and FIELD_THREE = VALUE ...
Instead of greater-than/less-than, you use + and - after the value. You can also use other comparisons as stated in the documentation. DMQL, as you might imagine, will be your biggest learning curve for querying RETS.
But to skip ahead of that, let’s look at the query in the example: ((112=2012-06-01\T00:00:00+),(178=ACT)) (pretend my PHP generated this date). Because there’s a + at the end, that means 112, the field for Last Update Date, has to have a value of June 1, 2012 or more recent. The other value, 118, requests that Status has to be ACT (Active; on the market). You can see other possible values with GetLookupValues(). We’re asking for Active listings that have been updated since June 1, 2012. As a sidenote, from personal experience I’ve found that listings older than that are usually forgotten/mistakes.
In the other options array, you can see the Format. COMPACT-DECODED is a good readout for information, and I would advise leaving that. You can also choose with which fields you’d like the RETS server to respond with Select (in this example I’m asking for the ID, address, city, state, MLS#, Update Date, and # of Photos), as well as the number of results to Limit (20). Count can be set to:
0No record count, data1Record Count + data2Record Count, no data
Obviously, 1 works best for most situations.
Running that script will result in something that looks like this:
Array ( [sysid] => 41268688 [49] => 800 N TAMIAMI TRL # 1205 [112] => 2012-09-28T10:28:01 [175] => A3966487 [9] => 10 [2302] => SARASOTA [2304] => Florida ) Array ( [sysid] => 40701173 [49] => 6500 MIDNIGHT PASS RD # 204 [112] => 2012-10-24T15:25:53 [175] => A3961874 [9] => 11 [2302] => SARASOTA [2304] => Florida ) ...
Success! You now have all the resources you need to take that data and do something great with it.
At this point, if all you need is the data, then you’re set. In case you want to download photos, here’s the last step:
Example 4: Downloading Photos with GetObject()
Downloading photos is surprisingly painless, after everything else you’ve been through. As you might imagine, photos aren’t part of a database query (nor should they be!), but are a separate request. Unfortunately, photos must be requested property-by-property, but this, for once, seems like a sane way to order things. Open example4.php. You’ll find this code:
$sysid = '12345678';
$n = 1;
$dir = 'photos';
if(!is_dir($dir)) mkdir($dir); // Remember: this can only make one directory at a time
$photos = $rets->GetObject('Property', 'Photo', $sysid);
foreach($photos as $photo) {
file_put_contents($dir.'/'.$n.'.jpg', $photo['Data']);
$n++;
}
$rets->FreeResult($photos);
$rets->Disconnect();
At this stage I don’t really see much to comment on, other than the fact that you have to use the sysid (from Example 3) of the property you want. It then returns an array of image file data, which can be saved with the good-ol’-fashioned file_put_contents(). The $n variable is simply a naming convention I use, and can be replaced with any naming structure you’d like.
In case you’re wondering, the only other data in the image array is:
Content-ID(same as thesysidyou gave it)Content-Type(image/jpegevery time I’ve checked)Object-ID(a number, usually 1, 2, 3, etc. but I depend on my own count for this)Success(0or1—if you got this message, wouldn’t it always be1?)
One thing I had to learn the hard way: on standard fat32 servers (read: most servers), you can only have 32,000 files and/or folders in any folder. This means if you have over 32,000 listings (many MLSs do), you can’t simply give each listing its own photo directory in the same master folder on your server; you have to break it up. For example, on Galleon, I had to take a listing like A2308004 out of the img/A2308004 directory and put the images into img/A/230/8004. So where there used to be 32,000 folders in img, now there are a maximum of 26 (since each MLS number starts with a letter). End of story: don’t have too many subfolders inside one folder. You will break it.
Conclusion
Obviously, this was just a primer into getting started with RETS. I didn’t go into practical usage scenarios, such as different ways to query listings, and how to store that data (like in a CSV, etc.). I don’t assume to know what you’re doing with it, but you should be in pretty good shape to take that data and place it any where you please. Your best bet from here would be to study more up on DMQL, and see what clever things other people are doing with RETS.
If you have any RETS-based project you’re working on, let me know! I’d be glad to hear about it.
Tips
- Consult the RETS Documentation. It’s terrible, but all you’ve got.
- P-A-R-S-E. Parse all the data you collect. These are all hand-entered by thousands of realtors, and are wrought with misspellings and case differences. Some aren’t preventable, but a lot can be managed.
- Ask your MLS which RETS version they’re using. This won’t affect a whole lot, but you might run into a specific version problem later with your code. You can set the version with
$rets->AddHeader("RETS-Version", "RETS/1.5");(1.5 is default for PHRETS). - Look into whether your MLS supports Unlimited Key Index. Typically, RETS servers will only serve a maximum number of listings if you ask for every field, but there are ways to grab all listings at once if you limit your query to only a few fields.
Seriously: go look at the awesome RETS DocumentationGo trudge through the River of Death (AKA the “new” RETS documentation).