Don't use count() when you don't need to return the exact number of rows
When you don't need to return the exact number of rows use:
1
DataFrame inputJson = sqlContext.read().json(...);
2
if (inputJson.takeAsList(1).size() == 0) {...}
3
4
or
5
6
if (inputJson.queryExecution.toRdd.isEmpty()) {...}
Copied!
instead of:
1
if (inputJson.count() == 0) {...}
Copied!
1
def isEmpty(): Boolean = withScope {
2
partitions.length == 0 || take(1).length == 0
3
}
Copied!
Last modified 11mo ago
Copy link