If this were 1998 and Janelle Monáe just released her music video, “Pynk,” she’d be sitting at the top of MTV’s Total Request Live every day for months.
Alas, it’s 2018 and we no longer have Carson Daly to guide our spiritual music journey. Which is why I’m here to ask you all to quietly bow down to Janelle Monáe next album, Dirty Computer, scheduled to drop April 27.
The album will apparently come with a 50-minute Afrofuturist film featuring black, brown, queer, and trans people under threat from an authoritarian government trying to scrape their memories.
According to a new profile in The New York Times Magazine, the film will feature Monáe as a “deviant” hiding from a totalitarian government and on the run to find her love interest (which just so happens to be Tessa Thompson, Monáe’s rumored partner). The government is on a mission to delete Monáe’s memories, as well as the memories of her fellow deviants (made up of people of color, queer folks, etc). Each video in the film is intended to reflect one of those missing moments
A new Monáe’s album would be relevant always, but it’s especially salient at this political moment in history. The Trump administration has come under fire for multiple acts of erasure: LGTBQ people will be removed from the next census, for example, putting services to the community under jeopardy. Both the DOJ and the DOE have dramatically weakened their department’s civil rights enforcement divisions.
It’s devastating, so pardon if we drown our sorrows in this image of Monáe a synchronized pussy pants militia.
Update: Apple has acknowledged the issue and is working on it. Statement and workaround below.
Wow, this is a bad one. On Macs running the latest version of High Sierra — 10.13.1 (17B48) — it appears that anyone can log in just by putting “root” in the user name field. This is a huge, huge problem. Apple will fix it probably within hours, but holy moly. Do not leave your Mac unattended until this is resolved.
The bug is most easily accessed by going to Preferences and then entering one of the panels that has a lock in the lower left-hand corner. Normally you’d click that to enter your user name and password, which are required to change important settings like those in Security & Privacy.
No need to do that any more! Just enter “root” instead of your user name and hit enter. After a few tries, it should log right in. There’s no need to do this yourself to verify it. Doing so creates a “root” account that others may be able to take advantage of if you don’t disable it.
The bug appears to have been first noticed by Lemi Orhan Ergin, founder of Software Craftsman Turkey, who noted it publicly on Twitter.
Needless to say, this is incredibly, incredibly bad. Once you log in, you’ve essentially authenticated yourself as the owner of the computer. You can add administrators, change critical settings, lock out the current owner, and so on. Do not leave your Mac unattended until this is resolved.
So far this has worked on every preference panel we’ve tried, and when I used “root” at the login screen it immediately created and pulled up a new user with system administrator privileges. It didn’t work on a 10.13 (17A365) machine, but that one is also loaded up with AOL bloatware — sorry, Oath bloatware — which may affect things.
Apple offered the following statement:
We are working on a software update to address this issue. In the meantime, setting a root password prevents unauthorized access to your Mac. To enable the Root User and set a password, please follow the instructions here: https://support.apple.com/en-us/HT204012. If a Root User is already enabled, to ensure a blank password is not set, please follow the instructions from the ‘Change the root password’ section.
You can find Directory Utility via the instructions in that link, but you can also hit command-space now to open Spotlight and just type it in. Once it opens, click the lock and enter your password and then under the Edit menu you’ll have the option to change the root password. It looks like this:
Anything’s better than nothing, which is the password the root user has now, but make it strong just in case.
We hope Apple has a fix soon because even though this workaround exists, we can’t be sure of the extent of this particular flaw until Apple takes a look. No one should leave their Mac unattended until this is resolved.
Alibaba Group Holding Ltd. kicked off its annual Singles’ Day shopping bonanza, logging $8.6 billion in sales within the first hour.
Shoppers from at least 192 countries and regions swarmed the e-commerce giant to scoop up discounted lobster, iPhones and refrigerators, at a rate of as many as 256,000 transactions per second. The Chinese company hosted a star-studded gala enlisting tennis star Maria Sharapova and American rapper Pharrell Williams to pump sales. As the event got underway, 93 percent of transactions were done via mobile.
Analysts are expecting another record day on Alibaba’s platforms, with Citigroup Inc. predicting a 31 percent rise in transactions to 158 billion yuan ($23.8 billion). While that’s only half of last year’s growth rate, it still dwarfs other events such as Black Friday and Cyber Monday. Billionaire founder Jack Ma is using this year’s event as a testing ground for his plans to revamp China’s $4 trillion traditional retail sector with technology, an experiment that could help the behemoth gain an edge in China’s saturated retail market.
“The work that’s been done in the integration of offline and online, not just in terms of the technology integration, but the data and efficiencies for brands and the consumer through personalization has been enormous,” Alibaba President Mike Evans said in an interview on Bloomberg TV. “We see the impact of it in our day-to-day business.”
Nov. 11 emerged as a counter-cultural antidote to the sentimentality surrounding Valentine’s Day. It takes its name from the way the date is written numerically as 11/11, which resembles “bare branches,” a local expression for the unattached.
Now, it’s become an excuse for people to shop and binge on entertainment shows. Hangzhou-based Alibaba is using the occasion to test the limits of its cloud computing, delivery and payments units — businesses that could benefit from roping in traditional retailers as customers. To that end, Alibaba teams fanned out across the nation ahead of Nov. 11 to help outlets — some 600,000 mom-and-pop convenient stores and some 1,000 brands — upgrade their computer systems. Those retailers, many in prime city locations, will become delivery and storage centers.
To connect a 10th of China’s 6 million convenient stores to the internet, Alibaba uses an app called Ling Shou Tong, meaning “connect retail.” Convenience stores are provided suggestions on what to procure and how to display merchandise. The goods are shipped from dedicated Alibaba warehouses, obviating the middlemen they would otherwise have dealt with. In theory, that improves their profit.
The company is also converting 100,000 retail outlets into so-called smart stores. Brands including Levis and L’Oreal are taking part. If one shop runs short on certain inventory, customers can track availability at other locations. They can also get goods delivered to their home.
Rivalry With Amazon
It’s early days in Alibaba’s grand retail experiment, but if it works, it could deepen a lead over Jeff Bezos’ Amazon.com Inc. in the fragmented world of physical retail. Ma’s company spent billions buying into grocers, shopping malls and even department stores years before Amazon announced its $13.7 billion acquisition of Whole Foods Market Inc.
Alibaba is already starting to see the initiative trickle into the top line. Revenue from new retail — mainly its Hema supermarkets and Intime department store — more than quintupled in the September quarter. As with convenience stores, Alibaba is trying to franchise the Hema model that combines a supermarket, restaurant and fulfillment center in a single location on its technology platform.
The company recently bought control of the unprofitable delivery business Cainiao Smart Logistics Network Ltd. It oversees a coterie of more than a dozen shipping partners, orchestrating deliveries carried out by millions of people across more than 600 cities. Alibaba has said Cainiao expects 3 million people to handle packages during the 24-hour-period, also known as 11-11.
“On the back of 11-11, we will probably have to distribute north of 700 million packages,” Evans said. “That is a massive, massive number of packages that requires a robust logistics network both in China and outside of China, and we will continue to invest in that business, and by moving to a controlling position, we will be able to ensure the right degree of quality.”
Hillary Clinton demanded that the FBI immediately explain its decision to reopen an investigation into her use of private e-mails as secretary of state as a new poll shows the race was a virtual toss-up even before the bureaus bombshell.
The American people deserve to get full and complete facts, Clinton told reporters in Des Moines, Iowa, late Friday, hours after her presidential campaign was rocked by a letter from FBI Director James Comey telling lawmakers of the revived inquiry. If theyre going to be sending this kind of letter that is only going originally to Republican members of the House, they need to share whatever facts they have with the American people.
The politically explosive development came less than two weeks before the election on Nov 8, providing a boost to Republican nominee Donald Trump as most national polls showed him lagging behind. Comey said he cant say how long the review would take — raising the possibility that Clinton could go into Election Day with the new probe unresolved and still hanging over her campaign.
An ABC News/Washington Post tracking poll released on Saturday Clintons lead over Trump has shrunk to two points from as many as 12 points less than a week ago, as more Republicans say theyll turn out for Trump. The poll was conducted Oct 24-27, before the new e-mail cache was announced.
The Federal Bureau of Investigation acted after investigators unearthed new e-mails through a separate probe of Clinton aide Huma Abedins estranged husband, former Representative Anthony Weiner. The bureau is probing illicit messages that Weiner allegedly sent to a 15-year-old girl in North Carolina.
That inquiry gave it access to a computer believed to have been used by both Weiner and Abedin, according a U.S. official who asked not to be identified discussing a pending investigation. Now the FBI is looking into whether Abedins e-mails are work-related and whether they contain classified information.
The more than 1,000 e-mails included exchanges between Abedin and Clinton, the Washington Post reported, citing a law enforcement official it didnt identify.
Comey offered no such specifics in a letter to eight committee chairmen.
In connection with an unrelated case, the FBI has learned of the existence of emails that appear to be pertinent to the investigation, Comey wrote. I agreed that the FBI should take appropriate investigative steps designed to allow investigators to review these emails to determine whether they contain classified information.
Comey said in an internal FBI memo that he wouldnt normally tell congress about a continuing investigation, according to Fox News. He wrote that he felt obligated because he had testified that the probe was complete, and also it would be misleading to the American people were we not to supplement the record. Comey added that the bureau doesnt know “the significance of this newly discovered collection of emails.”
Trump celebrated the FBIs decision during an appearance with supporters in New Hampshire. This is bigger than Watergate, Trump told the crowd Friday afternoon, which began chanting lock her up after he told them about the probe.
Clinton dismissed the potential political impact of the FBI move. People a long time ago made up their minds about the e-mails, she said. I think thats factored into what people think and now theyre choosing a president.
Clintons campaign manager, John Podesta, slammed the timing of Comeys decision.
“It is extraordinary that we would see something like this just 11 days out from a presidential election, Podesta said in a statement. The Director owes it to the American people to immediately provide the full details of what he is now examining. We are confident this will not produce any conclusions different from the one the FBI reached in July, he said, referring to a federal probe of Clintons e-mail practices that concluded without charges.
U.S. stocks erased gains as markets again showed themselves sensitive to perceptions about Clintons prospects, with the S&P 500 Index falling 0.3 percent in New York. The Mexican peso dropped, a reflection of Trumps plan to renegotiate trade pacts with the country and reduce immigration.
Clinton had appeared to be cruising toward a dominant win in the election. She held an average four-point lead over Republican Trump in polls that include independent candidates as of Friday, according to Real Clear Politics.
Some recent polls have been far worse for Trump; the Associated Press said on Oct. 26 that its poll showed Clinton with a 14-point lead. The election projection site FiveThirtyEight.com assessed her odds of a win at 82 percent on Friday. Yet the ABC poll showed Republicans and Republican-leaning independents starting to fall in line as Election Day nears.
Trump, who previously had denounced the FBI and Justice Department for failing to pursue charges against Clinton, praised the agencies on Friday. I have great respect that the FBI and Department of Justice have the courage to right the horrible mistake that they made, Trump said. This was a grave miscarriage of justice that the American people fully understood.
House Speaker Paul Ryan, a Wisconsin Republican, said in a statement that This decision, long overdue, is the result of her reckless use of a private email server, and her refusal to be forthcoming with federal investigators.
When the original investigation was closed in July, Comey faulted Clinton and her aides for extremely careless handling of classified information, but said the evidence wasnt sufficient to warrant prosecution. Attorney General Loretta Lynch subsequently announced that no charges would be brought against any individuals within the scope of the investigation.
Comey on Friday gave lawmakers no indication in his letter about the importance of the new information. FBI press officials declined to comment.
Although the FBI cannot yet assess whether or not this material may be significant, and I cannot predict how long it will take us to complete this additional work, I believe it is important to update your committees about our efforts, Comey wrote.
This story originally appeared on Grist and is part of the Climate Desk collaboration.
If you’re like me, you’ve probably been ignoring the bitcoin phenomenon for years — because it seemed too complex, far-fetched, or maybe even too libertarian. But if you have any interest in a future where the world moves beyond fossil fuels, you and I should both start paying attention now.
Last week, the value of a single bitcoin broke the $10,000 barrier for the first time. Over the weekend, the price nearly hit $12,000. At the beginning of this year, it was less than $1,000.
If you had bought $100 in bitcoin back in 2011, your investment would be worth nearly $4 million today. All over the internet there are stories of people who treated their friends to lunch a few years ago and, as a novelty, paid with bitcoin. Those same people are now realizing that if they’d just paid in cash and held onto their digital currency, they’d now have enough money to buy a house.
That sort of precipitous rise is stunning, of course, but bitcoin wasn’t intended to be an investment instrument. Its creators envisioned it as a replacement for money itself—a decentralized, secure, anonymous method for transferring value between people.
But what they might not have accounted for is how much of an energy suck the computer network behind bitcoin could one day become. Simply put, bitcoin is slowing the effort to achieve a rapid transition away from fossil fuels. What’s more, this is just the beginning. Given its rapidly growing climate footprint, bitcoin is a malignant development, and it’s getting worse.
Cryptocurrencies like bitcoin provide a unique service: Financial transactions that don’t require governments to issue currency or banks to process payments. Writing in the Atlantic, Derek Thompson calls bitcoin an “ingenious and potentially transformative technology” that the entire economy could be built on — the currency equivalent of the internet. Some are even speculating that bitcoin could someday make the US dollar obsolete.
But the rise of bitcoin is also happening at a specific moment in history: Humanity is decades behind schedule on counteracting climate change, and every action in this era should be evaluated on its net impact on the climate. Increasingly, bitcoin is failing the test.
Digital financial transactions come with a real-world price: The tremendous growth of cryptocurrencies has created an exponential demand for computing power. As bitcoin grows, the math problems computers must solve to make more bitcoin (a process called “mining”) get more and more difficult—a wrinkle designed to control the currency’s supply.
Today, each bitcoin transaction requires the same amount of energy used to power nine homes in the US for one day. And miners are constantly installing more and faster computers. Already, the aggregate computing power of the bitcoin network is nearly 100,000 times larger than the world’s 500 fastest supercomputers combined.
The total energy use of this web of hardware is huge—an estimated 31 terawatt-hours per year. More than 150 individual countries in the world consume less energy annually. And that power-hungry network is currently increasing its energy use every day by about 450 gigawatt-hours, roughly the same amount of electricity the entire country of Haiti uses in a year.
That sort of electricity use is pulling energy from grids all over the world, where it could be charging electric vehicles and powering homes, to bitcoin-mining farms. In Venezuela, where rampant hyperinflation and subsidized electricity has led to a boom in bitcoin mining, rogue operations are now occasionally causing blackouts across the country. The world’s largest bitcoin mines are in China, where they siphon energy from huge hydroelectric dams, some of the cheapest sources of carbon-free energy in the world. One enterprising Tesla owner even attempted to rig up a mining operation in his car, to make use of free electricity at a public charging station.
In just a few months from now, at bitcoin’s current growth rate, the electricity demanded by the cryptocurrency network will start to outstrip what’s available, requiring new energy-generating plants. And with the climate conscious racing to replace fossil fuel-base plants with renewable energy sources, new stress on the grid means more facilities using dirty technologies. By July 2019, the bitcoin network will require more electricity than the entire United States currently uses. By February 2020, it will use as much electricity as the entire world does today.
This is an unsustainable trajectory. It simply can’t continue.
There are already several efforts underway to reform how the bitcoin network processes transactions, with the hope that it’ll one day require less electricity to make new coins. But as with other technological advances like irrigation in agriculture and outdoor LED lighting, more efficient systems for mining bitcoin could have the effect of attracting thousands of new miners.
It’s certain that the increasing energy burden of bitcoin transactions will divert progress from electrifying the world and reducing global carbon emissions. In fact, I’d guess it probably already has. The only question at this point is: by how much?
The Blockchain Explained
The blockchain. Everyone's talking about it. But what is it, how does it work, and what's it for?
Former Uber site reliability engineer Susan Fowler accused the company of rampant sexual harassment and human resources negligence in a blog post published today.
Its the latest in a series of eventsthat point to serious questions about Uberscompany culture.
Fowler claims that on her first day out of training, she was solicited for sex by a superior onan internal company chat thread. She then immediately captured screenshots of the messages and sent them to Ubers human resources department. In a healthy organization, such a problem would have been resolved quickly. ButFowler alleges that the harassment only continued, preventing her from moving up within the company.
Upper management told me that he was a high performer and they wouldnt feel comfortable punishing him for what was probably just an innocent mistake on his part, explained Fowler in her post.
Atthis point,Fowler says in her post that she was given a choice of remaining on the team and accepting, a poor performance review, or moving to a different team.
I was then told that I had to make a choice: (i) I could either go and find another team and then never have to interact with this man again, or (ii) I could stay on the team, but I would have to understand that he would most likely give me a poor performance review when review time came around, and there was nothing they could do about that, further explained Fowler.
Though shedidnt want to leave the role she felt she was best prepared to fill, she switched teams. Work continued, and whileFowler had settled into the new role she regularlyhad conversations with female employees who shared similar stories about HR negligence, even citing unacceptable experiences with the same superior who solicited her. Along with a number of her colleagues, Fowler met once again with HR to make the point that the experiences of harassment were epidemic. Fowler then says thatUber insistedthat the manager had only been accused of a single offense.
Amidchaotic internal politics,Fowler attemptedto transferto a different department, but the company blocked her request. Citing strong performance, she couldnt understand why her request had been denied.
I was told that performance problems arent always something that has to do with work, but sometimes can be about things outside of work or your personal life,’ addedFowler in her post.
She ultimately decided to stay in the same roleuntil her next performance review. But the frustration continued with a second reassignment rejection and a further explanation that her review had been changed after the fact, and that she didnt show signs of an upward career trajectory. As a result, she was shut out of a company-sponsored Stanford computer science graduate program for high-achievers.
Aside from these claims,Fowler also describes in her post a culture of pervasive sexism telling the story of an employee who refused to order jackets in womens sizing because they cost more. No matter how many complaints she brought forth, HR insinuatedthat shewas the common denominator in all of her complaints. Fowler says she wasthreatened andintimidated in an effort to stop her fromreporting transgressions to HR.
In response to Fowlers post,Uber CEOTravisKalanick promisedto investigate the claims. In a statement to Axios,Kalanickmade a point to draw a dichotomy between the accused behavior and what the CEO believes is core to the companys culture:
I have just read Susan Fowlers blog. What she describes is abhorrent and against everything Uber stands for and believes in. Its the first time this has come to my attention so I have instructed Liane Hornsey our new Chief Human Resources Officer to conduct an urgent investigation into these allegations. We seek to make Uber a just workplace and there can be absolutely no place for this kind of behavior at Uber and anyone who behaves this way or thinks this is OK will be fired.
Uber board member and media mogul Arianna Huffington said in a tweet that she would conduct an independent investigation into the matter. Huffington even released her email address in an effort to make it easier for those with information to come forward.
Just talked w/ Travis & as a representative of Uber’s Board I will work w/Liane to conduct a full independent investigation starting now 1/2
Uber is no stranger to being in the negative spotlight when it comes to company culture not just with interpersonal relationships, but in its bigger business model and how it interfaces in the competitive environment for transportation services. In 2014, one of its senior executives (who is still at the company)tolda room full of journalists that Uber runsopposition research on its critics. One of the critics singled out had been very outspoken (along with many others) about how Uber does not take passenger safety seriously enough.
We still dont know the number of female engineers at Uber because the company hasnt been transparent about its hiring Jesse Jackson has made it his priority to change this. But even ifKalanick werentcomplicit,Fowlers experience could speak to how Uber values employee performance with respect to ethics and decency.
We have reached out to Uber and CEO TravisKalanick and will update this post when we hear back.
Google is releasing a new TensorFlow object detection API to make it easier for developers and researchers to identify objects within images. Google is trying to offer the best of simplicity and performance the models being released today have performed well in benchmarking and have become regularly used in research.
The handful of models included in the detection API include heavy duty inception-based convolutional neural networks and streamlined models designed to operate on less sophisticated machines a MobileNets single shot detector comes optimized to run in real-time on a smartphone.
Earlier this week Google announced its MobileNets family of lightweight computer vision models. These models can handle tasks like object detection, facial recognition and landmark recognition.
Todays smartphones dont possess the computational resources of larger scale desktop and server-based setups, leaving developers with two options. Machine learning models can run in the cloud, but that adds latency and requires an internet connection non-starters for a lot of common use cases. The alternative approach is simplifying the models themselves, making a trade-off in the interest of more ubiquitous deployment.
Of course Googles public cloud offerings give it differentiated positioning with respect to both Facebook and Apple, and its not new to delivering computer vision services at scale vis–vis its Cloud Vision API.
Todays TensorFlow object detection API can be found here. Google wants to make it extra easy to play with and implement so the entire kit comes prepackaged with weights and a Jupyter notebook.
Vision Mercedes-Maybach 6 concept is nearly 6 meters long.
People seem to have forgotten that Mercedes resurrected the Maybach brand as an all-new suffix to Mercedes’ most luxurious models, like the Mercedes-Maybach S600. To remedy that, it’s created a new luxury car of monumental proportions: the Vision Mercedes-Maybach 6 concept.
You might not be able to tell from the renderings, but it is truly monumental. To emphasize that point, let me give you a few of its specs straight away. First off, it’s 18-feet-eight-inches long (that’s almost 6 meters long, hence the 6 in its moniker).
Despite its gargantuan size, its all-electric, all-wheel drive powertrain will push it from 0 to 62 mph in less than four seconds on the way to its electronically limited top speed of 155 mph. Now, it can accelerate that quickly thanks to its 80-kilowatt-hour battery pack that puts out 738 horsepower.
Although it’s big and powerful, it’s still relatively efficient, going by the U.S. Environmental Protection Agency efficiency standards, it’s rated at a 200-mile range on a single charge. What’s more, thanks to fast-charging tech, it can receive charge that lasts for 62 miles of driving in just five minutes. The Vision Mercedes-Maybach 6 includes both wireless inductive charging and wired charging.
To access its “technoid” (Mercedes’ word, not mine) interior, occupants flip open the gullwing doors and drop into the coupe’s luxurious and techie cabin.
There, they’ll find a blend of old-world luxury, like open-pore elm wood floors and soft, quilted leather. These classic features are juxtaposed by high-tech highlights like a windshield that doubles as both a window and transparent digital display. The Vision Mercedes-Maybach 6 projects driving data and navigation information onto the windshield. And it’s operated with gesture controls.
The tech isn’t just functional, designers added some fashion, too. The transparent center tunnel in between the driver and passenger projects a visual representation of the flow of electricity from the powertrain to the wheels. The harder the driver pushes the accelerator pedal, the more energy is shown flowing. Granted, it’s just a digital approximation. But it’s still a cool idea.
Of course, such a high-tech car is also autonomous. With a push of a button, the driver can designate driving duties to the car. After all, there’s nothing more luxurious than having a computer do the driving for you.
Intriguingly, the new Vision Mercedes-Maybach 6 has the same wheelbase as the Vision Tokyo concept and wheels inspired by the brand’s Concept IAA. This is no coincidence. These three vehicles likely show a realistic vision of Mercedes’ future vehicle plans, from a self-driving lounge to an efficient, morphing luxury sedan to an extremely luxurious sports EV coupe.
With the Vision Mercedes-Maybach 6 concept, Mercedes proves that although the future of mobility will be governed by automated driving and extremely efficient designs, it doesn’t mean it has to be boring.
On Saturday morning, the white stone buildings on UC Berkeleys campus radiated with unfiltered sunshine. The sky was blue, the campanile was chiming. But instead of enjoying the beautiful day, 200 adults had willingly sardined themselves into a fluorescent-lit room in the bowels of Doe Library to rescue federal climate data.
Like similar groups across the country—in more than 20 cities—they believe that the Trump administration might want to disappear this data down a memory hole. So these hackers, scientists, and students are collecting it to save outside government servers.
But now theyre going even further. Groups like DataRefuge and the Environmental Data and Governance Initiative, which organized the Berkeley hackathon to collect data from NASAs earth sciences programs and the Department of Energy, are doing more than archiving. Diehard coders are building robust systems to monitor ongoing changes to government websites. And theyre keeping track of whats been removedto learn exactly when the pruning began.
Tag It, Bag It
The data collection is methodical, mostly. About half the group immediately sets web crawlers on easily-copied government pages, sending their text to the Internet Archive, a digital library made up of hundreds of billions of snapshots of webpages. They tag more data-intensive projectspages with lots of links, databases, and interactive graphicsfor the other group. Called baggers, these coders write custom scripts to scrape complicated data sets from the sprawling, patched-together federal websites.
Its not easy. All these systems were written piecemeal over the course of 30 years. Theres no coherent philosophy to providing data on these websites, says Daniel Roesler, chief technology officer at UtilityAPI and one of the volunteer guides for the Berkeley bagger group.
One coder who goes by Tek ran into a wall trying to download multi-satellite precipitation data from NASAs Goddard Space Flight Center. Starting in August, access to Goddard Earth Science Data required a login. But with a bit of totally legal digging around the site (DataRefuge prohibits outright hacking), Tek found a buried link to the old FTP server. He clicked and started downloading. By the end of the day he had data for all of 2016 and some of 2015. It would take at least another 24 hours to finish.
The non-coders hit dead-ends too. Throughout the morning they racked up 404 Page not found errors across NASAs Earth Observing System website. And they more than once ran across empty databases, like the Global Change Data Centers reports archive and one of NASAs atmospheric CO2 datasets.
And this is where the real problem lies. They don’t know when or why this data disappeared from the web (or if anyone backed it up first). Scientists who understand it better will have to go back and take a look. But meantime, DataRefuge and EDGI understand that they need to be monitoring those changes and deletions. Thats more work than a human could do.
So theyre building software that can do it automatically.
Later that afternoon, two dozen or so of the most advanced software builders gathered around whiteboards, sketching out tools theyll need. They worked out filters to separate mundane updates from major shake-ups, and explored blockchain-like systems to build auditable ledgers of alterations. Basically its an issue of what engineers call version control—how do you know if something has changed? How do you know if you have the latest? How do you keep track of the old stuff?
There wasnt enough time for anyone to start actually writing code, but a handful of volunteers signed on to build out tools. Thats where DataRefuge and EDGI organizers really envision their movement goinga vast decentralized network from all 50 states and Canada. Some volunteers can code tracking software from home. And others can simply archive a little bit every day.
By the end of the day, the group had collectively loaded 8,404 NASA and DOE webpages onto the Internet Archive, effectively covering the entirety of NASAs earth science efforts. Theyd also built backdoors in to download 25 gigabytes from 101 public datasets, and were expecting even more to come in as scripts on some of the larger datasets (like Teks) finished running. But even as they celebrated over pints of beer at a pub on Euclid Street, the mood was somber.
There was still so much work to do. Climate change data is just the tip of the iceberg, says Eric Kansa, an anthropologist who manages archaeological data archiving for the non-profit group Open Context. There are a huge number of other datasets being threatened with cultural, historical, sociological information. A panicked friend at the National Parks Service had tipped him off to a huge data portal that contains everything from park visitation stats to GIS boundaries to inventories of species. While he sat at the bar, his computer ran scripts to pull out a list of everything in the portal. When its done, hell start working his way through each quirky dataset.
UPDATE 5:00pm Eastern, 2/15/17: Phrasing in this story has been updated to clarify when changes were made to federal websites. Some data is missing, but it is still unclear when that data was removed.
It was December 2012, and Doug Burger was standing in front of Steve Ballmer, trying to predict the future.
Ballmer, the big, bald, boisterous CEO of Microsoft, sat in the lecture room on the ground floor of Building 99, home base for the companys blue-sky R&D lab just outside Seattle. The tables curved around the outside of the room in a U-shape, and Ballmer was surrounded by his top lieutenants, his laptop open. Burger, a computer chip researcher who had joined the company four years earlier, was pitching a new idea to the execs. He called it Project Catapult.